NTUA-SLP at SemEval-2018 Task 1: Predicting Affective Content in Tweets with Deep Attentive RNNs and Transfer Learning

• Twitter’s emojis, abbreviations, hashtags, slang, and other non-conventional forms make natural language processing more challenging.Source: §1 · p. 1
• Earlier sentiment systems relied on hand-crafted or sentiment-lexicon features fed to classifiers, requiring extensive expert feature engineering that struggles to keep pace with language evolution.Source: §1 · p. 1
• Neural networks for text classification had outperformed conventional approaches and could learn features from data while also using hand-crafted features; SemEval-2018 Task 1 included emotion classification and regression.Source: §1 · p. 2
• The system combines transfer learning from SemEval Task 4A with a word-level Bi-LSTM and deep self-attention, and visualizes salient tokens to aid interpretation.The pretrained model is fine-tuned for each task to address limited task-specific training data.Source: §1 · p. 2

2 Overview

• The pipeline trains embeddings, pretrains a model for sentiment analysis, then fine-tunes it for each affect task.The tasks cover emotion and valence intensity prediction, ordinal classification, and multi-label emotion classification.Source: §2 · p. 2

2.1 Data

• The system uses 550 million English tweets for embedding training and preprocessing statistics, and 61,854 sentiment-labeled tweets from SemEval 2017 Task 4A for pretraining.The source labels are positive, neutral, and negative.Source: §2.1 · p. 2

2.2 Word Embeddings

• Twitter-trained word2vec embeddings are augmented with 10 affective dimensions whose word ratings are estimated from manually annotated norms using semantic similarity and a linear model.The dimensions include valence, dominance, arousal, pleasantness, anger, sadness, fear, disgust, concreteness, and familiarity.Source: §2.2 · pp. 2–3

2.3 Preprocessing1

• Ekphrasis applies Twitter-specific tokenization, spell correction, normalization, hashtag segmentation, and annotation; normalization reduces vocabulary size without removing words.The processor uses statistics from the unlabeled Twitter dataset for prior probabilities in spell correction and segmentation.Source: §2.3 · p. 3

2.4 Neural Transfer Learning for NLP

• The model is pretrained on semantically similar Twitter sentiment data, then its BiLSTM and attention encoder initialize each task model with a task-specific output layer.The authors note that transfer beyond pretrained word vectors is less pervasive in NLP.Source: §2.4 · p. 4

2.5 Recurrent Neural Networks

• RNNs process words sequentially, preserving order and handling variable-length inputs; LSTMs and GRUs use gating to address gradient growth or decay over long sequences.The passage describes the final hidden state as a summary of the sequence.Source: §2.5 · p. 4

2.6 Self-Attention Mechanism

• Self-attention represents a sequence as a learned convex combination of all hidden states, rather than relying only on the final state; the figure compares regular and attentive RNNs.The learned weights indicate each hidden state's contribution to the sequence representation.Source: §2.6 · p. 4, Figure 3 · p. 4

3 Model Description

• For every subtask, the system pretrained a deep-attention BiLSTM on SA2017, replaced its final layer with a task-specific layer, and fine-tuned the whole network.Source: §3 · p. 4

3.1 Transfer Learning Model (TF)

• The model concatenates forward and backward word annotations, applies multilayer self-attention to weight informative words, and feeds the resulting representation to task-specific output layers.Output layers use a sigmoid for regression, softmax for ordinal classification, or 11 sigmoid outputs for multi-label classification.Source: §3.1 · p. 5

3.2 Fine-Tuning

• Full-network fine-tuning significantly outperformed freezing the pretrained encoder across all tasks.Both schemes replaced the pretrained model’s final layer with a task-specific layer.Source: §3.2 · p. 6

3.3 Regularization

• Regularization adds Gaussian noise and dropout and uses early stopping; the embedding layers remain fixed during fine-tuning.The authors explain that test-only words otherwise retain their initial embedding positions, which may no longer reflect their “true” emotion and lead to mis-classifications.Source: §3.3 · p. 6

4.1 Experimental Setup

• The models use Adam, gradient-norm clipping, class weights for imbalanced EI-oc and V-oc classes, and Bayesian optimization for hyperparameter tuning.The encoder has two 250-unit-per-direction BiLSTM layers and two self-attention layers; the embedding layer combines 300 word2vec and 10 affective dimensions.Source: §4.1 · p. 6

4.2 Experiments

• The setup compares BOW, N-BOW, and affective-feature N-BOW baselines with randomly initialized, frozen-transfer, and fine-tuned Bi-LSTMs across five subtasks.Neural results average 10 runs; Pearson correlation is used for four subtasks and Jaccard index for E-c.Source: §4.2 · p. 6, Table 2 · p. 7
• Neural models outperformed all baselines by a large margin; transfer learning helped most emotion-intensity subtasks but not E-c, and fine-tuning significantly improved results overall.For joy intensity, random initialization matched frozen transfer learning in EI-reg and EI-oc.Source: §4.2 · p. 7
• The mystery-set evaluation pairs sentences differing only in social context and measures per-class prediction differences with significance tests.The paper frames this as a test for statistically significant social bias; results are summarized in Table 3.Source: §4.2 · p. 7, Table 3 · p. 7

4.3 Attention visualizations

• Attention heat maps show emotion-salient words, often select emojis, and can highlight dependencies between distant words.The visualization covers eight example tweets, with two tweets per emotion.Source: §4.3 · p. 7, Figure 10 · p. 8

4.4 Competition Results

• 2/48 (EI-reg), 5/39 (EI-oc), 4/38 (V-reg), 8/37 (V-oc, tied for 6th and 7th), and 1/35 (E-c) were the official ranks.All subtasks used LSTM-TL-FT with the same pretrained model; the paper reports competitive results across them.Source: §4.4 · pp. 7–8

5 Conclusion

• The models achieved excellent results in single- and multi-label classification, but mixed results in emotion- and valence-intensity tasks.The conclusion presents these outcomes as the paper's overall task-level pattern.Source: §5 · p. 8
• Future work would revisit transfer learning with alternative approaches, including methods cited by the authors.The cited approaches are Felbo et al. (2017), Howard and Ruder (2018), and Hashimoto et al. (2016).Source: §5 · p. 8
• The authors propose adding character-level information to address out-of-vocabulary words and learn emotion-indicative syntactic and stylistic features.The proposed character-level direction cites Wieting et al. (2016), Labeau and Allauzen (2017), and Peters et al. (2018).Source: §5 · p. 8
• The authors make their pretrained word embeddings and model source code available to support reproducibility and further experimentation.The paper gives a project repository address for the code.Source: §5 · p. 8