Multimodal Sentiment Analysis To Explore the Structure of Emotions

1 INTRODUCTION

• Tumblr posts often pair images with text, making photographs and memes a valuable source for studying social-media communication.The authors aim to give social scientists tools to investigate how images are used on social networks.Source: §1 · p. 1
• Tumblr emotion tags such as “#happy” are treated as proxies for self-reported emotion, which the authors consider a psychologically meaningful label.The paper notes that self-report is the gold standard for measuring emotions in psychology.Source: §1 · p. 1
• Deep Sentiment combines fine-tuned Inception image features, GloVe word embeddings, a recurrent network, and a dense layer to predict emotion tags.Its softmax output gives a probability distribution over possible emotion word tags.Source: §1 · p. 1

2 RELATED WORK

• Visual sentiment analysis is challenging because image meaning can require abstraction and depends on cultural knowledge and subjective interpretation.Two people may use the same image to express different emotions.Source: §2 · pp. 1–2
• Earlier work analyzed visual emotion with semantic attributes or convolutional networks, while image-annotation research found gains from combining text and image features.The cited approaches include SentiBank, CNN-based binary sentiment classification, and image-annotation studies and.Source: §2 · p. 2
• Prior multimodal sentiment methods include non-negative matrix factorisation and latent correlations, and one study modeled text, visual, and emotional views of image tweets.Chen et al. reported that two thirds of participants added tweet images to enhance the text’s emotion.Source: §2 · p. 2
• Earlier social-media studies lacked large labeled image-text training sets, and polarity-focused text analysis may not reliably measure a writer’s latent emotion.The paper addresses both concerns with a large noisy-labeled Tumblr dataset and a goal of predicting emotional state rather than sentiment polarity.Source: §2 · p. 2

3 TUMBLR DATASET

• Tumblr posts commonly contain an image, text, and tags, and the paper treats emotion word tags as noisy proxies for the user’s state of mind.This operationalizes the labels as proxies for underlying emotional state rather than positive-versus-negative sentiment.Source: §3 · p. 2
• The dataset retains 15 emotions selected for high Tumblr frequency among emotions on the PANAS-X scale or Plutchik’s wheel.Posts were retrieved through Tumblr API queries for emotion tags.Source: §3 · p. 2
• Posts were collected backward from 2017 to 2011, but API limits prevented high-density emotions such as “happy” or “angry” from reaching as far back.Emotion words appearing in post text were removed to avoid giving the classifier an unfair advantage.Source: §3 · p. 2
• The dataset excludes posts with less than 50% English words and posts without images.Table 1 summarizes dataset statistics for posts from January 2011 to September 2017; the dataset cannot be redistributed, but code is available to replicate it.Source: §3 · p. 2, Table 1 · p. 3
• Example posts pair an “Optimistic” label with a hopeful message and a “Happy” label with a view at McWay Falls.The examples illustrate posts with associated emotion tags; they do not provide dataset-wide outcome statistics.Source: Figure 1 · p. 2, Figure 2 · p. 2

4.1 Visual analysis

• The visual pathway fine-tunes pretrained Inception representations, which the authors argue capture image colors and shape arrangements relevant beyond the original task.This relies on the hypothesis that images lie on a low-dimensional manifold.Source: §4.1 · p. 3

4.2 Natural language processing

• Image-only interpretation can be ambiguous, while accompanying text can clarify which emotion a post conveys.The paper gives an example where a caption changes the interpretation of an image.Source: §4.2 · p. 3, Figure 3 · p. 3

4.2.1 Word embedding.

• GloVe embeddings map words into a high-dimensional space intended to preserve semantic relationships, using Twitter-trained vectors because its writing style resembles Tumblr’s.The paper contrasts this with one-hot representations, where “dog” is no closer to “cat” than to “tree.”Source: §4.2.1 · p. 3

4.2.2 Sequence input.

• The text pathway preserves word order with recurrent modeling rather than averaging word vectors, which would lose order-dependent meaning such as negation.Posts are embedded as word sequences and fed to an LSTM; shorter posts are padded, longer posts are truncated to 50 words, and 76% contain fewer than 50 words.Source: §4.2.1 · p. 3, §4.2.2 · p. 3

4.3 Deep Sentiment: a multimodal neural network

• Deep Sentiment combines visual recognition and text analysis because text can change how an image’s emotion is interpreted.The paper names the proposed combined network architecture “Deep Sentiment.”Source: §4.3 · p. 4

4.3.1 Architecture.

• The architecture processes a 224×224×3 image into 256 features and text through a 1024-unit LSTM, then concatenates both modalities before dense and softmax layers.The softmax output is a probability distribution over the user’s emotional state.Source: Figure 4 · p. 4, §4.3.1 · p. 4

5 EVALUATION

• The comparison includes image-only and text-only models, plus random guessing that incorporates class prior probabilities.The text describes Table 2 as comparing these baselines with Deep Sentiment.Source: §5 · p. 4, Table 2 · p. 4
• Figures 5 and 6 compare the models’ train-accuracy and test-accuracy curves, respectively.The supplied captions identify Figure 5 as train accuracy and Figure 6 as test accuracy.Source: §5 · p. 4, Figure 5 · p. 4, Figure 6 · p. 4
• 80% train accuracy and 72% test accuracy: Deep Sentiment significantly outperforms the image model and slightly outperforms the text-only model.No validation set was used to tune hyperparameters.Source: §5 · p. 4, Table 2 · p. 4

6.1 Top words for each emotion

• The model’s emotion-specific word lists are sensible and include modern social-media terms; “woke” ranks for Scared and Amazed, while “phone” is outside LIWC’s emotion categories.“Woke” also appears in the top 20 for Calm.Source: §6.1 · p. 5

6.2 Clustering emotions

• Emotion clustering separates positive emotions from negative ones and, within negative emotions, groups low-arousal bored and sad apart from high-arousal angry, ashamed, annoyed, and disgusted.The authors describe these patterns as consistent with previous psychology findings.Source: §6.2 · p. 5

6.3 The circumplex model of emotion

• 28.8% of variance is explained by PC1, versus 16.4% by PC2 and 14.6% by PC3, evidence against a simple two-factor model.The circumplex model posits valence and arousal as two dimensions explaining emotion.Source: §6.3 · p. 5
• In the PCA visualization, emotion categories vary substantially across the first two components, although related emotions such as happy and love lie close together.The authors report that happy and love are also close in the dendrogram.Source: §6.3 · p. 5
• PC1 corresponds well to valence, separating happy and love from disgusted, ashamed, and annoyed; PC2 is not clearly arousal, and adding PC3 gives no clearer picture.Together, these observations provide further evidence against a simple two-factor model.Source: §6.3 · p. 5

6.4 Validation on the Open Affective Standardized Image Set (OASIS)

• The authors characterize Tumblr posts with user-selected emotion tags as richer emotional data than images rated only on valence and arousal.They describe the latter task as more similar to standard positive-versus-negative sentiment analysis.Source: §6.4 · p. 6
• OASIS ratings show high correlation between PC1 and valence but low correlation with arousal; PC2 and PC3 correlate with arousal, but also with valence.The authors interpret these mixed associations as further evidence against two factors explaining emotion.Source: Table 4 · p. 6, §6.4 · p. 6

• Because public social-media posts are shaped by self-presentation, the study cannot rule out bias and may capture expressed or performed emotion rather than emotion as truly experienced.Source: §7 · p. 7
• Future work will test the model on other image-text stimuli datasets used in psychology and compare its accuracy with human judges.Source: §7 · p. 7
• The authors also plan to investigate daily and day-of-week trends in emotion.Source: §7 · p. 7

• The appendix includes examples of Tumblr posts tagged Disgusted, Sad, Angry, and Surprised.Source: A APPENDIX · p. 8
• Table 5 lists the top 11–20 words for each emotion.Source: Table 5
• Appendix figures show the emotion correlation matrix, PCA projections on dimensions 1 and 3 and dimensions 2 and 3, and the PCA explained-variance plot.Source: Figure 10 · p. 8, Figure 11 · p. 9, Figure 12 · p. 9