Source-linked AI summary
Does syntax matter? A strong baseline for Aspect-based Sentiment Analysis with RoBERTa
Junqi Dai, Hang Yan, Tianxiang Sun, Pengfei Liu, Xipeng Qiu
TL;DR
The paper investigates whether PTMs contain sufficient task-relevant syntax for aspect-level sentiment classification, rather than relying on parser-provided dependency trees. It induces and compares trees before and after ALSC fine-tuning, finding that fine-tuned RoBERTa learns sentiment-word-oriented structure and that a simple RoBERTa model achieves SOTA or near-SOTA results across six datasets and four languages.
Problem
The paper asks whether PTMs’ implicit syntactic information can support strong ALSC models without relying on parser-provided dependency trees.
Method
The study induces trees from pretrained and ALSC-fine-tuned BERT and RoBERTa using Perturbed Masking, then compares them with parser trees in tree-based ALSC models.
Results
A base RoBERTa model with an MLP achieves SOTA or near-SOTA performance on all six ALSC datasets across four languages.
Takeaways & Limitations
Fine-tuned RoBERTa implicitly incorporates task-oriented syntactic information, with induced trees oriented toward connecting aspects to sentiment words.
Takeaways & Limitations
Adding external trees to RoBERTa-based ALSC models brings no significant improvement and can cause performance declines, possibly because the external and implicit trees are difficult to reconcile.
Abstract
from arXiv · showhide
Aspect-based Sentiment Analysis (ABSA), aiming at predicting the polarities for aspects, is a fine-grained task in the field of sentiment analysis. Previous work showed syntactic information, e.g. dependency trees, can effectively improve the ABSA performance. Recently, pre-trained models (PTMs) also have shown their effectiveness on ABSA. Therefore, the question naturally arises whether PTMs contain sufficient syntactic information for ABSA so that we can obtain a good ABSA model only based on PTMs. In this paper, we firstly compare the induced trees from PTMs and the dependency parsing trees on several popular models for the ABSA task, showing that the induced tree from fine-tuned RoBERTa (FT-RoBERTa) outperforms the parser-provided tree. The further analysis experiments reveal that the FT-RoBERTa Induced Tree is more sentiment-word-oriented and could benefit the ABSA task. The experiments also show that the pure RoBERTa-based model can outperform or approximate to the previous SOTA performances on six datasets across four languages since it implicitly incorporates the task-oriented syntactic information.
1 Introduction
The paper asks whether PTM-induced syntax can replace parser-provided dependency structure for ALSC, and whether fine-tuning adapts that structure to sentiment. It finds that fine-tuned RoBERTa provides task-oriented trees, while a simple RoBERTa model reaches strong results without explicit trees.
- ABSA predicts sentiment polarity for each aspect, and this paper focuses specifically on aspect-level sentiment classification.
- The paper asks whether PTM-induced trees outperform parser-provided dependency trees in tree-based ALSC models.
- Fine-tuning adapts PTMs’ implicit trees for ALSC: fine-tuned PTM-induced trees outperform their pre-fine-tuning counterparts, with fine-tuned RoBERTa even surpassing parser trees.
- A base RoBERTa model with an MLP achieves SOTA or near-SOTA performance on all six ALSC datasets across four languages.
- The authors study induced trees from PTMs and fine-tuned PTMs, finding fine-tuned RoBERTa trees outperform other examined trees.
- Fine-tuned PTM-induced trees tend to orient more toward sentiment words, directly connecting aspect terms with sentiment adjectives.
2 Related Work
Prior ALSC research moved from handcrafted syntactic features toward neural models and dependency-tree integration. Related work also examined PTMs for implicit syntactic knowledge and graph-based dependency modeling.
- Early ALSC systems relied on manually designed syntactic features, while later neural models sought to avoid handcrafted features.
- Neural networks and graph neural networks substantially improved dependency parsing quality and enabled richer dependency-tree modeling.
- Some ALSC methods convert dependency trees into aspect-oriented trees, while others exploit tree-based token distances.
- PTM probing studies report that models such as BERT implicitly capture some dependency-tree structure.
3 Method
The method induces trees from BERT and RoBERTa using Perturbed Masking, then evaluates them in representative tree-based ALSC architectures. It also uses a simple RoBERTa fine-tuning model and compares alternative tree structures.
- PTM-induced trees: The paper selects BERT and RoBERTa as representative PTMs and applies Perturbed Masking to induce trees without additional parameters.
- BERT and RoBERTa: BERT and RoBERTa use Transformer backbones, with representations formed through layered self-attention, feed-forward networks, and normalization.
- Perturbed Masking: Perturbed Masking estimates how much masking token x_j changes token x_i’s representation, using Euclidean distance as the impact value.
- Perturbed Masking: Repeating pairwise impact estimation produces an impact matrix, from which a dependency tree is decoded using algorithms such as Chu–Liu/Edmonds.
- Tree-based ALSC models: The study evaluates induced and parser-provided trees with three representative tree-based ALSC models, including ASGCN, PWCN, and RGAT.
- Tree-based ALSC models: RGAT converts a dependency tree into an aspect-oriented tree rooted at the aspect, encoding syntactic tags or tree-based distances.
4 Experimental Setup
Experiments cover six ALSC datasets across four languages and compare parser, PTM-induced, fine-tuned PTM-induced, and chain-based trees. Fine-tuned models use a simple MLP-based RoBERTa architecture, with layer selection performed on development data.
- Datasets: The experiments use six benchmark datasets across four languages, including English restaurant, laptop, and Twitter datasets.
- Tree structures: For each dataset, the study obtains trees from dependency parsers, pretrained BERT and RoBERTa, fine-tuned BERT and RoBERTa, plus left-chain and right-chain constructions.
- Fine-tuning: Fine-tuned BERT and RoBERTa trees are induced after ALSC fine-tuning with a simple MLP to introduce few additional parameters.
- Tree selection: Trees are induced from each PTM layer and evaluated on a development set; the 11th layer performs best and is used throughout the experiments.
- Implementation details: In RGAT experiments, induced trees lack syntactic tags, so virtual tags are assigned uniformly, which slightly damages performance.
5 Experimental Results
Experiments compare parser-provided, PTM-induced, and fine-tuned PTM-induced trees for ALSC, alongside pure RoBERTa models and quantitative structural analyses. Fine-tuned RoBERTa induces the strongest tree-based results, while base RoBERTa alone reaches SOTA or near-SOTA performance across the evaluated datasets.
- ALSC Performance with Different Trees: Fine-tuned RoBERTa induced trees achieve the best results on all datasets, averaging 1.1% higher accuracy than dependency trees.The comparison uses tree-based ALSC models on English datasets, with accuracy and macro-F1 reported across three runs.
- ALSC Performance with Different Trees: Dependency-tree models usually outperform models using off-the-shelf BERT or RoBERTa induced trees.The PTM-induced trees tend toward one-sided neighboring dependencies, resembling left-chain and right-chain baselines.
- ALSC Performance with Different Trees: Fine-tuning improves induced-tree performance: FT-RoBERTa yields a 1.56% average accuracy gain over RoBERTa induced trees.The same improvement trend appears between BERT induced trees and FT-BERT induced trees.
- Analysis: Fine-tuning reduces neighboring connections and produces a more diverse dependency pattern in FT-RoBERTa induced trees.Base BERT/RoBERTa induced trees average over 70% neighboring relations, whereas fine-tuning lowers this proportion.
- Analysis: FT-RoBERTa has the lowest Aspects-sentiment Distance, indicating shorter distances between aspects and sentiment words.Compared with off-the-shelf PTM-induced trees, fine-tuned PTM-induced trees have lower AsD; Twitter parsing results are not fully explained by these metrics.
- Comparison between ALSC models: RoBERTa with an MLP layer achieves SOTA or near-SOTA performance on the six ALSC datasets, while adding tree structures provides no concrete improvement.The authors report especially strong improvement on Laptop14 and limited gains over the RoBERTa-MLP model from similar RoBERTa-based tree models.
6 Conclusion
The paper finds that fine-tuning helps RoBERTa induce sentiment-word-oriented trees, while a fine-tuned RoBERTa model with an MLP achieves strong ALSC performance without tree integration.
- Fine-tuning on ALSC forces PTMs to learn more sentiment-word-oriented trees that benefit Glove-based ALSC models.
- The fine-tuned RoBERTa with an MLP achieves SOTA or near-SOTA ALSC results across six datasets and four languages.
- The paper analyzes parser-provided dependency trees alongside trees induced from PTMs for ALSC.
A Experiments on non-English Datasets
This section introduces the paper’s experiments on non-English datasets.
- The experiments in this section concern non-English datasets.
- The section provides experimental details for the non-English setting.
- The non-English experiments are presented as a distinct experimental section.
A.1 Datasets
The study uses Dutch, French, and Spanish restaurant-review datasets from SemEval-2016 Task 5, comparing parser-provided, induced, and chain-based tree structures.
- Datasets: The experiments use Dutch, French, and Spanish restaurant-review datasets from SemEval-2016 task 5.The dataset languages correspond to their names, and samples with conflicting polarities or NULL aspect terms are removed.
- Tree construction: Five tree types are obtained using dependency parsing or Perturbed Masking on pretrained and fine-tuned language models.The induced-tree variants include BERT, RoBERTa, FT-BERT, and FT-RoBERTa trees.
- Baselines: Left-chain and right-chain structures serve as additional baselines, linking each word to its previous or next word.
A.3 Implementation Details
The implementation evaluates tree-based ALSC models with multiple tree structures alongside fine-tuned pretrained language models, using fixed pretrained embeddings and standardized settings.
- Experimental setup: Tree-based ALSC models are evaluated with different trees, alongside fine-tuning experiments for pretrained language models.
- Model configuration: Tree-based models use 300-dimensional pretrained embeddings that remain fixed to avoid overfitting.Other parameters are initialized from the original models.
- Model configuration: RGAT reproductions assign uniform virtual relations because induced trees do not provide relation labels.
- Optimization: Fine-tuning uses batch size b = 32, dropout rate d = 0.1, learning rate µ = 2e-4, and AdamW with default settings.
- Tree selection: All induced trees used in the experiments come from the 11th layer.
A.4.1 ALSC Performance with Different Trees
Fine-tuned RoBERTa-induced trees deliver the strongest tree-based ALSC results, while fine-tuning improves induced-tree performance and standalone PLM models remain highly competitive.
- FT-RoBERTa Induced Tree leads to the best results on all non-English datasets.It also produces more stable F1 scores than the alternatives.
- PLM-induced trees slightly outperform parser-provided dependency trees on non-English datasets, where parser and corpus quality can constrain dependency-tree performance.The observed sensitivity is attributed to the quality of non-English parsers and corpora.
- Fine-tuning substantially improves ALSC performance when trees are induced from PLMs.The comparison between RoBERTa Induced Tree and FT-RoBERTa Induced Tree supports the benefit of task-specific adaptation.
- Fine-tuned XLM-R with an MLP achieves SOTA or near-SOTA results on non-English datasets.The comparison uses a set of top high-performing ALSC models.
- The sentiment-word analysis uses 50 frequent sentiment words from Amazon-2 to calculate Aspects-sentiment Distance on English datasets.The words are taken from Tian et al. (2020).