Source-linked AI summary
Exploiting BERT for End-to-End Aspect-based Sentiment Analysis
Xin Li, Lidong Bing, Wenxuan Zhang, Wai Lam
TL;DR
E2E-ABSA requires jointly detecting aspects and their sentiments, motivating evaluation of contextualized pre-trained embeddings for this sequence-labeling task. The paper couples BERT with simple neural baselines and finds BERT-based models superior, including robustness to overfitting, while standardizing development-set model selection.
Problem
E2E-ABSA jointly detects aspect terms or categories and their sentiments, whereas several related ABSA settings assume aspects are given and require expensive fine-grained annotation.
Method
The paper couples BERT embeddings with various neural sequence-labeling models, including linear, recurrent, self-attention, and conditional-random-field downstream layers.
Results
BERT-based models outperform existing works, with even BERT-Linear surpassing works without BERT; the models also show robustness to overfitting.
Takeaways & Limitations
The study provides a BERT-based benchmark for E2E-ABSA by combining simple baselines with consistent hold-out development-set model selection.
Takeaways & Limitations
The paper focuses on the aspect-term-level E2E-ABSA setting, while settings assuming given aspects are less practical because fine-grained annotation is expensive.
Abstract
from arXiv · showhide
In this paper, we investigate the modeling power of contextualized embeddings from pre-trained language models, e.g. BERT, on the E2E-ABSA task. Specifically, we build a series of simple yet insightful neural baselines to deal with E2E-ABSA. The experimental results show that even with a simple linear classification layer, our BERT-based architecture can outperform state-of-the-art works. Besides, we also standardize the comparative study by consistently utilizing a hold-out validation dataset for model selection, which is largely ignored by previous works. Therefore, our work can serve as a BERT-based benchmark for E2E-ABSA.
1 Introduction
ABSA benchmarks sentiment toward explicit aspects or implicit categories, while E2E-ABSA jointly detects aspect terms or categories and their sentiments. The paper motivates contextualized BERT embeddings and standardized validation for this more practical sequence-tagging task.
- ABSA problem settings: ABSA discovers users’ sentiment toward explicitly mentioned aspect terms or implicit aspect categories in user-generated text.SemEval ABSA challenges provide widely used benchmark datasets with review sentences and gold sentiment annotations.
- ABSA problem settings: E2E-ABSA jointly detects aspect terms or categories and their corresponding aspect sentiments as a sequence-tagging problem.Unlike original ABSA, E2E-ABSA does not assume the aspects are given.
- Motivation: Task-agnostic Word2Vec- or GloVe-initialized embedding layers have contributed to a bottleneck in accuracy and F1 improvements for original ABSA models.The paper motivates context-aware embeddings pre-trained with deep LSTM or Transformer models as a potential improvement.
- Paper scope: The paper explores BERT’s modeling power for E2E-ABSA through simple neural baselines, either fine-tuning task-specific components or using BERT as a feature extractor.The study explicitly avoids aiming at a task-specific architecture.
- Practical motivation: E2E-ABSA is more practical than settings that assume given aspects because manual annotation of fine-grained aspect mentions or categories is expensive.The paper contrasts E2E-ABSA with original ABSA and AOWE, which assume the aspects are given.
- Evaluation practice: The comparative study consistently uses a hold-out development dataset for model selection, a practice ignored in most existing ABSA works.This standardization is presented as part of the paper’s benchmark contribution.
2 Model
The model treats aspect-term E2E-ABSA as sequence labeling: BERT produces contextualized token representations, and task-specific layers predict sentiment-aware tags. The paper compares linear, recurrent, self-attention, and CRF-based downstream designs.
- BERT as embedding layer: BERT processes the input token sequence through L Transformer layers to produce contextualized representations for downstream prediction.The representations incorporate sentence-level information rather than a single context-independent vector per token.
- Task formulation: E2E-ABSA is formulated as sequence labeling, with tags jointly encoding aspect boundaries and positive, negative, or neutral sentiment.The tag set includes B-, I-, E-, and S- prefixes plus O for tokens outside aspects.
- Downstream models: The downstream E2E-ABSA layer is evaluated with linear classification, GRU-based recurrent modeling, self-attention variants, and a linear-chain CRF.These designs differ in how they model task-specific token representations or tag dependencies.
- Downstream models: The GRU design adds layer normalization when calculating gates because directly applying an RNN to Transformer outputs may produce unstable training.The model uses reset, update, and new gates before applying a softmax prediction layer.
- Downstream models: The CRF models dependencies between adjacent tags through a transition matrix and decodes the globally highest-scoring tag sequence.Its sequence-level softmax ranges over all possible tag sequences, with decoding performed by Viterbi search.
3 Experiment
Experiments on two SemEval-derived review datasets evaluate BERT-based E2E-ABSA models against existing sequence-labeling approaches. BERT-Linear already outperforms non-BERT works, BERT models remain robust to overfitting, and task-specific fine-tuning is essential.
- Dataset and Settings: Experiments use two SemEval-derived review datasets and select models on development-set micro-averaged F1.The study uses LAPTOP and REST, with model selection every 100 steps after 1000 training steps.
- Dataset and Settings: BERT-based models are compared with tailor-made E2E-ABSA systems and LSTM-CRF sequence-labeling models.The comparison includes existing E2E-ABSA models and competitive sequence-labeling baselines.
- Main Results: BERT-Linear outperforms existing works without BERT, while stronger downstream layers further improve performance.The result suggests contextualized BERT representations alleviate context independence in the linear layer, and added context benefits sequence modeling.
- Over-parameterization Issue: BERT-GRU, BERT-TFM, and BERT-CRF show stable development-set F1 scores during extended training on REST.Their F1 scores do not decrease much through 3000 training steps, indicating robustness to overfitting.
- Finetuning BERT or Not: Fixed general-purpose BERT representations perform worse than models using task-specific fine-tuning.The authors conclude that fine-tuning is essential for exploiting BERT on the downstream task.
4 Conclusion
The paper evaluates BERT embeddings for E2E-ABSA through varied neural models and extensive experiments. BERT-based models capture aspect sentiment, remain robust to overfitting, and benefit from fine-tuning.
- Conclusion: The study couples BERT embeddings with various neural models for E2E-ABSA and evaluates them on two benchmark datasets.The experiments assess BERT-based models' effectiveness on capturing aspect-based sentiment.
- Conclusion: BERT-based models demonstrate superiority in capturing aspect-based sentiment and robustness to overfitting.These conclusions summarize the reported experimental findings.