Source-linked AI summary
Adapt or Get Left Behind: Domain Adaptation through BERT Language Model Finetuning for Aspect-Target Sentiment Classification
Alexander Rietzler, Sebastian Stabinger, Paul Opitz, Stefan Engl
TL;DR
Aspect-Target Sentiment Classification benefits from pretrained language models, while real-world use can involve data from distributions different from training data. The paper adapts BERT through domain-specific language-model finetuning and evaluates task performance, cross-domain transfer, and prediction errors, achieving state-of-the-art restaurant performance and improved cross-domain results.
Problem
ATSC has substantial potential for pretrained-language-model methods, but in-domain evaluation does not always reflect real-world prediction on previously unseen data.
Method
The approach finetunes BERT's language model on a target domain before supervised ATSC training in a source domain, using masked-token and next-sequence objectives.
Results
The restaurant model achieves new state-of-the-art performance, while cross-domain adaptation significantly improves unadapted models and can outperform an in-domain BERT-base model.
Takeaways & Limitations
Cross-domain evaluation indicates that ATSC transfers well between laptops and restaurants, and domain-specific language-model finetuning improves robustness across domains.
Takeaways & Limitations
Remaining errors include neutral examples confused with positive or negative sentiment and predictions influenced by sentiment toward different aspect-targets.
Abstract
from arXiv · showhide
Aspect-Target Sentiment Classification (ATSC) is a subtask of Aspect-Based Sentiment Analysis (ABSA), which has many applications e.g. in e-commerce, where data and insights from reviews can be leveraged to create value for businesses and customers. Recently, deep transfer-learning methods have been applied successfully to a myriad of Natural Language Processing (NLP) tasks, including ATSC. Building on top of the prominent BERT language model, we approach ATSC using a two-step procedure: self-supervised domain-specific BERT language model finetuning, followed by supervised task-specific finetuning. Our findings on how to best exploit domain-specific language model finetuning enable us to produce new state-of-the-art performance on the SemEval 2014 Task 4 restaurants dataset. In addition, to explore the real-world robustness of our models, we perform cross-domain evaluation. We show that a cross-domain adapted BERT language model performs significantly better than strong baseline models like vanilla BERT-base and XLNet-base. Finally, we conduct a case study to interpret model prediction errors.
1 Introduction
The paper focuses on ATSC, the aspect-target variant of ABSA, and investigates BERT language-model finetuning for domain adaptation and performance improvement.
- Aspect-Target Sentiment Classification: ATSC determines the sentiment polarity associated with an extracted aspect-target, such as (“dumplings”, “positive”).It follows Aspect-Target Extraction, which detects terms referring to aspects.
- Motivation: The authors approach ATSC with BERT because pretrained language models have become effective bases for downstream NLP tasks.They focus on exploiting BERT’s architecture for ATSC.
- Contributions: The paper analyzes how the amount of BERT language-model finetuning affects ATSC performance.This extends prior work by investigating the relationship between finetuning steps and end-task performance.
- Contributions: The study reports new state-of-the-art performance on the SemEval 2014 restaurants dataset through BERT language-model finetuning.The contribution concerns how to exploit finetuning for the restaurants domain.
- Contributions: Cross-domain adaptation is evaluated by finetuning BERT on a target-domain corpus, training supervised ATSC models in the source domain, and also combining both datasets.The analysis compares transfer between laptop and restaurant domains.
2 Related Works
Prior ATSC work improves performance through specialized architectures or transfer across tasks and domains; this paper instead targets cross-domain transfer through self-supervised BERT finetuning.
- Research directions: Related work separates architecture-focused ATSC methods from approaches that transfer knowledge across related tasks or domains.This frames the paper’s contribution relative to two broad research directions.
- Neural architectures: ATSC research introduced architectures such as Memory Networks, Attention Encoder Networks, and graph convolutional networks to address limited training data and model target-context relations.These methods explicitly model context, attention, or dependencies between sentiment words and aspect-targets.
- Knowledge transfer: Knowledge transfer has been used across domains or related tasks to compensate for insufficient ATSC training examples.Examples include transfer from aspect-category classification, document-level sentiment classification, and question answering.
- Paper’s distinction: Unlike prior task-transfer methods, this paper transfers knowledge across domains by self-supervised finetuning of BERT’s language model.The proposed direction adapts the language model rather than transferring from a different source task.
3 Methodology
The methodology uses BERT as the foundation for a two-step ATSC pipeline: domain-specific language-model adaptation followed by supervised end-task training, with input reduction for error interpretation.
- Overall procedure: The ATSC pipeline first finetunes pretrained BERT on a domain-specific corpus self-supervisedly, then trains it supervisedly on the ATSC task.This is the paper’s central modeling procedure.
- Evaluation design: The experiments evaluate end-task training and domain-specific finetuning combinations for both in-domain and cross-domain generalization.The methodology includes comparisons across training and adaptation domains.
- Interpretation: Input reduction is used to interpret neural NLP model predictions and examine ATSC errors.The paper applies this interpretation method after describing the model and training combinations.
- BERT: BERT uses deeply bidirectional sequence representations built from contextualized representations, transformers, and language-model pretraining.Its pretraining is followed by end-to-end finetuning on downstream tasks.
- BERT: BERT’s pretraining objective combines masked language modeling with next-sequence prediction.Masked language modeling predicts masked tokens, while next-sequence prediction models whether one sequence follows another.
3.2 BERT Language Model Finetuning
BERT language-model finetuning is performed as an intermediate, domain-specific adaptation step, and the paper studies how its training duration affects ATSC performance.
- Finetuning procedure: Domain-specific language-model finetuning is algorithmically equivalent to pretraining and precedes supervised ATSC training.The procedure starts from BERT weights and adapts them using domain-specific corpora.
- Finetuning procedure: The study investigates how end-task ATSC performance depends on the number of language-model finetuning steps.This extends earlier work on using domain-specific finetuning as an intermediate step.
- Input representation: Language-model finetuning represents input as two sequences formatted “[CLS] sA [SEP] sB [SEP]”.[CLS] is used for downstream classification, while [SEP] separates the sequences.
Masked Language Model Objective
BERT is adapted for ATSC by learning domain-relevant language patterns through masked-token prediction, then using the resulting representation for three-way sentiment classification of a sentence and target pair.
- Masked-token prediction: Domain-specific finetuning masks tokens in review text so BERT learns to predict contextually appropriate words.The example contrasts Wikipedia’s likely “input” with the review-domain opinion word “amazing.”
- ATSC input: BERT’s downstream ATSC input combines a tokenized sentence and its target as “[CLS] s [SEP] t [SEP]”.This reformats the sentence-target pair for BERT sequence-pair classification.
- ATSC classifier: The classifier maps the final [CLS] representation to a three-class distribution over positive, negative, and neutral polarity.A fully connected layer with three output neurons is followed by softmax.
3.4 Domain Adaptation through Language Model Finetuning
The paper frames domain adaptation as improving robustness when training and test data come from different distributions. Its procedure finetunes BERT toward a target domain before supervised ATSC transfer.
- Motivation: In-domain evaluation can misrepresent real-world performance because deployed models encounter previously unseen data.In-domain tests share the training distribution, whereas applications may not.
- Domain adaptation: Domain Adaptation evaluates generalization across domains and adapts the model toward the target domain when supervised task data is available only for a source domain.The paper identifies this as a special case of Transductive Transfer Learning.
- Transfer framework: The framework treats ATSC as the transfer task and BERT language-model finetuning as domain adaptation.The language model is finetuned on a domain-specific corpus before task transfer.
- Evaluation design: Language-model finetuning, ATSC training, and testing can use Restaurants, Laptops, or their union, yielding nine evaluation scenarios grouped into four categories.Testing is restricted to Restaurants or Laptops, while joint training combines both datasets.
In-Domain Training
The in-domain settings train ATSC on one domain and evaluate it on that same domain or on the other domain, distinguishing matched-domain from cross-domain testing.
- In-Domain Training: In-domain training evaluates an ATSC model on the test set from the same domain used for training.The setting is represented as DLM →T →T, with T equal to Laptops or Restaurants.
- In-Domain Training: Cross-domain training evaluates an ATSC model trained on one domain against the test set from the other domain.The source and target domains differ, represented as DLM →S →T.
Cross-Domain Adaptation
Cross-domain adaptation finetunes the language model on the target domain before training on the source domain and testing on the target, while joint training combines both domains.
- Cross-Domain Adaptation: Target-domain adaptation is the cross-domain variant in which DLM = T, represented as T →S →T.The paper expects this configuration to optimize cross-domain performance.
- Cross-Domain Adaptation: Joint cross-domain training combines source and target datasets, then evaluates independently on each target-domain test set.This scenario is represented as DLM →(S ∪T) →T.
3.5 Input Reduction for Model Interpretation
Input reduction interprets model predictions by identifying words that most support the original label, then uses the reduced inputs to examine errors and model differences. In ATSC, aspect-target phrases remain fixed while other words are removed until the prediction changes.
- Input reduction identifies a document subset whose words contribute most to a model prediction.
- The method illustrates test-set predictions to investigate classification errors and qualitative differences between models and baselines.
- Word importance equals the original-label probability with a word present minus the probability after removing that word.
- The procedure repeatedly removes the least important word until the model changes its predicted label.
- Aspect-target phrases are exempt from removal so the reduced input retains context for the targeted aspect.
4 Experiments
Experiments evaluate domain-specific BERT finetuning for ATSC across training iterations, in-domain, cross-domain, and joint-domain settings. Results show significant gains, new restaurant-domain state of the art, and generally strong cross-domain generalization, while neutral cases remain difficult.
- RQ1: Finetuning dynamics: Domain-specific BERT finetuning significantly improves ATSC performance; restaurants improve immediately, laptops after about 10 million sentences, with no significant gains beyond roughly 17 million.Different runs show high variance, so the authors average 9 runs to measure differences reliably.
- RQ3: Cross-domain adaptation: 2.2% absolute accuracy improvement on laptops and 3.6% on restaurants over BERT-base show that target-domain language-model finetuning supports cross-domain adaptation.The cross-domain setting finetunes BERT on the target domain and trains the ATSC classifier in the source domain.
- RQ3: Cross-domain adaptation: ATSC generalizes cross-domain with about a 2-3% accuracy drop relative to in-domain training.The authors suggest that aspect-target sentiment syntax and sentiment polarity knowledge are sufficient in most cases.
- Joint-domain training: Combining both training datasets improves performance on both test sets, with especially strong Macro-F1 for the joint model.The authors associate this pattern with better neutral-class classification, especially for laptops, although they present this as a possible explanation.
- Error analysis: Remaining errors disproportionately involve neutral examples, including confusion with positive restaurant sentiment, negative laptop sentiment, sentiment from another aspect-target, and absence statements.The analysis also identifies a handful of examples the authors believe may have incorrect ground-truth labels.
5 Conclusion
The paper uses domain-specific BERT language-model finetuning before downstream ATSC training, analyzing finetuning steps and evaluating in-domain and cross-domain performance. It reports new state-of-the-art restaurant performance, strong cross-domain transfer, and gains from target-domain adaptation over unadapted models.
- The experiments first finetune pre-trained BERT on domain-specific text, then train it on the downstream ATSC classification task.
- The study analyzes how the number of domain-specific BERT finetuning steps relates to end-task performance.
- The SemEval 2014 Task 4 restaurants model achieves new state-of-the-art performance after optimizing the use of BERT language-model finetuning.
- Cross-domain evaluation finds that ATSC transfers well between the laptops and restaurants domains in this setup.
- Target-domain language-model adaptation significantly improves over unadapted models, with one adapted model outperforming in-domain BERT-base.
- The authors identify further work in applying domain-specific finetuning to XLNet-base and testing cross-domain behavior with a hotel domain.