PaperlayerView exact revision on arXiv

Source-linked AI summary

SentiBERT: A Transferable Transformer-Based Architecture for Compositional Sentiment Semantics

Da Yin, Tao Meng, Kai-Wei Chang

arXiv:2005.04114v4cs.CL

TL;DR

SentiBERT addresses the challenge of modeling compositional sentiment semantics, especially negation and contrast in complex sentences. It combines contextualized representations with binary constituency trees and achieves improved phrase-level sentiment classification, while transferring learned composition to related sentiment and emotion tasks.

  • Problem

    Compositional sentiment semantics, including negation and contrast, remains difficult to model in complex sentences.

  • Method

    SentiBERT combines contextual information with explicit binary constituency-tree structure to model semantic composition and evaluates transfer from SST to related tasks.

  • Results

    SentiBERT improves significantly over recursive networks and base BERT on SST phrase-level sentiment classification and transfers well to sentiment and emotion tasks.

  • Takeaways & Limitations

    The results support SentiBERT’s effectiveness and transferability for compositional sentiment semantics, including contrastive relations and broader task settings.

  • Takeaways & Limitations

    The paper identifies that child-node aggregation models can overlook rich contextual associations.

Abstract

from arXiv · show

We propose SentiBERT, a variant of BERT that effectively captures compositional sentiment semantics. The model incorporates contextualized representation with binary constituency parse tree to capture semantic composition. Comprehensive experiments demonstrate that SentiBERT achieves competitive performance on phrase-level sentiment classification. We further demonstrate that the sentiment composition learned from the phrase-level annotations on SST can be transferred to other sentiment analysis tasks as well as related tasks, such as emotion classification tasks. Moreover, we conduct ablation studies and design visualization methods to understand SentiBERT. We show that SentiBERT is better than baseline approaches in capturing negation and the contrastive relation and model the compositional sentiment semantics.

1 Introduction

SentiBERT targets the difficulty of modeling compositional sentiment, especially when negation and contrast change phrase sentiment. It combines contextualized BERT representations with constituency structure and transfers learned phrase-level composition across sentiment-related tasks.

  • 1 Introduction: Negation and contrast can change sentiment between child and parent phrases, making compositional sentiment difficult to model in complex sentences.Figure 1 highlights sentiment changes caused by negation and contrast, including the contrastive conjunction “but.”
  • 1 Introduction: SentiBERT combines contextualized BERT representations with recursive constituency structure to capture compositional sentiment semantics.Training uses masked language modeling for context and phrase-node sentiment prediction for composition.
  • 1 Introduction: SentiBERT significantly improves over recursive networks and base BERT on phrase-level sentiment classification.
  • 1 Introduction: The architecture uses a BERT encoder, an attention-based semantic composition module, and a phrase-level sentiment predictor.The composition module attends first to covered tokens and then to phrase children.
  • 1 Introduction: SentiBERT trained on SST transfers to Twitter sentiment analysis, emotion intensity classification, and contextual emotion detection.

2 Related Work

Prior sentiment classifiers include feature-based, recursive, convolutional, recurrent, and pretrained language-model approaches. SentiBERT extends contextualized representation modeling by leveraging binary constituency parse trees for compositional sentiment.

  • 2 Related Work: Sentiment analysis has been approached with feature-based, recursive, convolutional, recurrent, and pretrained language-model methods.
  • 2 Related Work: SentiBERT leverages binary constituency parse trees alongside contextualized representations to model compositional sentiment semantics.
  • 2 Related Work: Semantic composition methods have also been applied to semantic relatedness and sememe knowledge beyond sentiment analysis.

3 Model

SentiBERT builds phrase representations from contextualized tokens and constituency structure through two levels of attention, then predicts phrase sentiment. Its training combines masked language modeling with phrase-node supervision and supports downstream transfer.

  • 3 Model: SentiBERT contains BERT, an attention-based semantic composition module, and phrase and sentence sentiment predictors.The architecture is illustrated as three modules.
  • 3 Model: The semantic composition module first attends to tokens covered by each phrase and then attends to the phrase and its children.This two-level mechanism combines contextualized representation with hierarchical constituency structure.
  • 3 Model: Phrase representations begin with contextualized token representations corresponding to each phrase’s constituent span.For phrase i, the covered tokens range from s_ti through e_ni.
  • 3 Model: Child-level attention computes weights over a phrase and its left and right children, then combines their representations into a refined phrase representation.The refined representation uses weighted contributions from both children and the phrase itself.
  • 3 Model: SentiBERT jointly uses masked language modeling and phrase-node prediction to learn contextual information and phrase-level sentiment composition.Phrase-node prediction is trained with cross-entropy loss.
  • 3 Model: A pretrained SentiBERT model can initialize downstream models in transfer-learning settings.

4 Experiments

SentiBERT is evaluated on phrase- and sentence-level sentiment tasks, transfer tasks, and emotion classification benchmarks. Results compare its contextualized, tree-guided architecture with recursive networks, pretrained-language-model baselines, and ablations.

  • 4.1 Experimental Settings: The experiments use SST phrase and sentence variants plus Twitter Sentiment Analysis, Emotion Intensity Classification, and EmoContext for transfer evaluation.SST-phrase predicts five sentiment classes for non-terminal constituents, while SST-5 predicts sentence sentiment.
  • 4.2 Effectiveness of SentiBERT: SentiBERT and SentiBERT with RoBERTa substantially outperform corresponding variants and networks built only on constituency trees.The comparison is reported for SST-phrase and SST-5 averaged accuracies across five runs.
  • 4.3 Transferability of SentiBERT: SentiBERT achieves competitive sentence-level performance across SST-3, Twitter Sentiment Analysis, and related sentiment classification tasks.It outperforms vanilla BERT and XLNet on tasks including SST-3 and Twitter Sentiment Analysis.
  • 4.2 Effectiveness of SentiBERT: SentiBERT outperforms its version without BERT, while its contextualized variants outperform corresponding vanilla BERT and RoBERTa baselines.These comparisons support both pretrained contextual representation and syntactic guidance as important components.
  • 4.3 Transferability of SentiBERT: SentiBERT trained on SST transfers to emotion tasks, achieving the best results on Emotion Intensity Classification and EmoContext.The reported metrics are averaged Pearson correlation for Emotion Intensity Classification and F1 for EmoContext.

5 Analysis

The analysis evaluates SentiBERT’s compositional sentiment modeling across structural difficulty, negation, contrastive relations, interpretability, and reduced phrase-level supervision. SentiBERT consistently outperforms baselines, with larger advantages on harder cases and useful transfer under limited supervision.

  • 5.1 Semantic Compositionality: SentiBERT performs better than baselines across local and global difficulty levels, with larger gaps as sentiment-switching difficulty increases.Local difficulty counts switches between a phrase and its children, whereas global difficulty counts switches across the entire constituency tree.
  • 5.1 Semantic Compositionality: At local difficulty 2, SentiBERT exceeds BERT w/ Tree-LSTM by about 7% accuracy and BERT w/ Mean pooling by 15%.The gap between SentiBERT and baselines increases with difficulty, supporting structural guidance and attention over hierarchical representations.
  • 5.2 Negation and Contrastive Relation: SentiBERT performs best as negation-word counts increase, and its advantage over baselines grows with negation difficulty.The evaluation groups sentences by the number of negation words and measures accuracy on their constituency predictions.
  • 5.2 Negation and Contrastive Relation: For ‘X but Y’ triple-lets, SentiBERT outperforms other BERT variants by about 1% when all three phrase sentiments must be correct.The evaluation targets sentences where the left and right parts have different sentiments.
  • 5.3 Case Study: Attention visualizations show SentiBERT focuses on sentiment-bearing adjectives, switches around negation, and emphasizes the part after ‘but’ in contrasting sentences.The visualized weights cover left child, phrase, and right child representations in the constituency hierarchy.
  • 5.4 Amount of Phrase-level Supervision: Using 30%–50% of SST phrase labels yields competitive SST-5 and SST-3 results, while 70%–80% supports competitiveness with XLNet on Twitter sentiment without phrase-level supervision.The study varies phrase-label availability from 0% to 100% before downstream fine-tuning.

6 Conclusion

The conclusion presents SentiBERT as a contextualized, syntactically guided architecture for compositional sentiment semantics. Experiments support its effectiveness, transferability, interpretability, and potential under reduced supervision.

  • 6 Conclusion: SentiBERT combines contextual information with explicit syntactic guidance to capture compositional sentiment semantics.The architecture uses a BERT component, an attention-based semantic composition module, and phrase- and sentence-level sentiment predictors.
  • 6 Conclusion: Experiments demonstrate SentiBERT’s effectiveness and transferability across sentiment and related emotion tasks.The conclusion summarizes the reported empirical findings without specifying individual benchmark values.
  • 6 Conclusion: Further analyses indicate that SentiBERT is interpretable and remains promising with less supervision.The paper also identifies extending the architecture to other applications involving phrase-level annotations as future work.

A.1 Details of Correlation Computation in Attention Networks

This appendix describes the correlation computation used in SentiBERT’s attention networks. It specifies the activation, parameterization, and separate parameters for the two attention layers.

  • A.1 Details of Correlation Computation in Attention Networks: The attention network computes a correlation between vectors a and b before assigning attention weights.The appendix introduces the displayed correlation expression as the computation used by the attention mechanism.
  • A.1 Details of Correlation Computation in Attention Networks: The correlation computation uses SeLU with α = 4, and the two attention-network layers do not share parameters.These are implementation choices specified for the attention computation.

A.2 Details of Downstream Tasks

The appendix defines the downstream sentiment and emotion benchmarks used to assess transferability. It specifies their label spaces, evaluation metrics, and benchmark-statistics tables.

  • A.2 Details of Downstream Tasks: The downstream evaluation includes SST-2/3, Twitter Sentiment Analysis, Emotion Intensity Ordinal Classification, and EmoContext.These tasks cover sentence sentiment, tweet sentiment, emotion intensity, and emotion classification in dialogue.
  • A.2 Details of Downstream Tasks: SST-3 uses three sentiment classes, while Twitter Sentiment Analysis predicts positive, negative, or neutral sentiment.SST-3 derives its labels by converting the five SST-5 phrase-level classes into three groups.
  • A.2 Details of Downstream Tasks: Emotion Intensity Classification assigns one of four intensity classes and reports averaged Pearson correlation across happiness, sadness, anger, and fearness.The metric averages Pearson correlation over the four emotion subtasks.
  • A.2 Details of Downstream Tasks: EmoContext classifies the last sentence of a two-turn dialogue using F1 over happy, sad, and angry, excluding ‘others’.The task evaluates contextual emotion detection in conversational text.
  • A.2 Details of Downstream Tasks: The appendix points to benchmark-statistics and node-distribution tables covering datasets, difficulty, negation, and token-node prediction.The cited tables summarize benchmark and analysis data rather than reporting new task definitions.

A.3 Details of Analysis Part

The analysis examines how nodes and sentences are distributed across local difficulty, global difficulty, and negation-word dimensions. These distributions are reported separately in Tables 6, 7, and 8.

  • A.3 Details of Analysis Part: Table 6 reports node and sentence distributions by local difficulty.
  • A.3 Details of Analysis Part: Table 7 reports node and sentence distributions by global difficulty.
  • A.3 Details of Analysis Part: Table 8 reports node and sentence distributions in terms of negation words.

A.4 Incorporating Token Node Prediction

The model combines token-node and phrase-node prediction objectives to model compositional sentiment semantics. Adding token-level sentiment information slightly lowers performance, possibly because token and phrase sentiment reflect different objectives.

  • A.4 Incorporating Token Node Prediction: Token-node and phrase-node prediction objectives are combined to model compositional sentiment semantics.
  • A.4 Incorporating Token Node Prediction: Results drop slightly when token-level sentiment information is additionally incorporated.
  • A.4 Incorporating Token Node Prediction: The authors attribute the performance drop possibly to inconsistency between compositional phrase sentiment and lexicon-dependent token sentiment objectives.
Loading 2005.04114v4…