CLMLF:A Contrastive Learning and Multi-Layer Fusion Method for Multimodal Sentiment Detection

• Complementary text-image signals can clarify sentiment: an image reveals positive intent absent from declarative text, while ruins in another image deepen negative sentiment.Source: §1 · p. 1
• The Transformer-Encoder MLF uses multi-head self-attention across text tokens and image patches to align and fuse token-level features, while multiple layers deepen abstraction.Source: §1 · p. 2
• The authors argue that concatenation, gating, attention, and memory-based methods offer limited fusion because different modality vector spaces make text-image alignment difficult.Source: §1 · pp. 1–2
• CLMLF achieves better performance than several baseline models on MVSA-Single, MVSA-Multiple, and HFM.Source: §1 · p. 2
• LBCL and DBCL use sentiment labels and data augmentation to learn common sentiment-related multimodal features and improve model performance.Source: §1 · p. 2

2.1 Overview

• The overall architecture comprises a multi-layer fusion module and a multi-task learning module with sentiment classification and both contrastive-learning tasks.Source: §2.1 · p. 2

2.2 Multi-Layer Fusion Module

• The multi-layer fusion module projects ResNet image features into the text dimension, encodes the image sequence, fuses it with text using a Transformer, and applies attention to obtain a multimodal representation.The fusion Transformer aligns and combines token-level text and image features.Source: §2.2 · p. 3

2.3 Sentiment Classification

• The sentiment-classification task feeds the multimodal representation to a fully connected layer and softmax, using cross-entropy loss.Source: §2.3 · p. 3

2.4 Label Based Contrastive Learning

• Label-based contrastive learning treats same-sentiment examples in a batch as positives and examples with different sentiment labels as negatives.The task calculates a contrastive loss from the batch labels and loss matrix.Source: §2.4 · p. 4

2.5 Data Based Contrastive Learning

• The data-based contrastive task uses augmented text-image examples to encourage invariant, sentiment-relevant features despite changes in surface form.Text is augmented with back-translation, while images use RandAugment.Source: §2.5 · p. 4

2.6 Model Training

• The label-based and data-based contrastive losses are added to the total loss as regularization, with coefficients balancing the training losses.Source: §2.6 · p. 5

3.1 Dataset

• Experiments use MVSA-Single, MVSA-Multiple, and HFM; MVSA splits are 8:1:1, with preprocessing following Xu and Mao (2017) and Cai et al. (2019), respectively.The MVSA datasets contain Twitter text-image pairs labeled with a single sentiment; HFM preprocessing follows Cai et al. (2019).Source: §3.1 · p. 5

3.2 Implementation Details

• CLMLF uses BERT-base and ResNet-50 encoders, AdamW with a 2e-5 learning rate, and batch sizes of 32, 64, and 128 for MVSA-Single, MVSA-Multiple, and HFM.Both contrastive-loss weights are 1.0; experiments use four NVIDIA 3090 GPUs. PyTorch and HuggingFace Transformers are used for implementation.Source: §3.2 · p. 5

3.3 Compared Methods

• Unimodal baselines include CNN, Bi-LSTM, TGNN, and fine-tuned BERT for text, plus OSDA and fine-tuned ResNet for images.The cited models are CNN (Kim, 2014), Bi-LSTM, TGNN (Huang et al., 2019), BERT (Devlin et al., 2019), OSDA, and ResNet (He et al., 2015).Source: §3.3 · p. 5
• Multimodal baselines span attention, co-memory, graph, concatenation, hierarchical fusion, and decomposition-relation models.They include MultiSentiNet, HSAN (Xu, 2017), Co-MN-Hop, MGNNS, Schifanella et al. (2016), MMSD, and D&R Net.Source: §3.3 · p. 5

4.1 Overall Result

• CLMLF is competitive with strong baselines across three datasets; multimodal models outperform unimodal ones, while image-only sentiment is weakest and gains are smaller on binary HFM.The authors suggest image sentiment features are sparse and noisy, and attribute HFM’s smaller improvement over BERT to its binary classification setting versus three-class MVSA-Single.Source: §4.1 · p. 6

4.2 Ablation

• The complete CLMLF model performs best in the ablation comparison; multi-layer fusion improves performance, and adding both contrastive tasks improves it further.The authors interpret the contrastive tasks as helping the model learn common sentiment features and separate data with different sentiments.Source: §4.2 · p. 6, Table 3

4.3 Influence of MLF Layer

• The selected fusion/image layer combinations were 3-2, 4-2, and 5-1 across the datasets, and HFM benefits more from text than images.The layer analysis indicates that text and image contributions differ by dataset; the authors therefore use more text-related than image-related transformer layers in the MLF module.Source: §4.3 · pp. 6–7, Figure 3 · p. 7

4.4 Case Study

• In two case-study examples, image context helps CLMLF correctly identify sentiment that text-only BERT misclassifies.One text refers to a positive image, while another appears negative from text alone but is a joke expressing positive sentiment when the image is considered.Source: §4.4 · p. 7, Figure 4 · p. 7

4.5 Visualization

• Attention visualization shows that the model aligns text keywords with corresponding image regions at token level, including when “lady” refers to only half a face.The visualization uses the first attention head of the final Transformer-Encoder layer in the Multi-Layer Fusion module.Source: §4.5 · p. 7, Image Text CLMLF BERT · p. 7, Figure 5 · p. 7

Image Text CLMLF BERT

• On MVSA-Single, CLMLF’s representations separate positive from negative sentiment more clearly and cluster neutral examples better than BERT’s.The authors use t-SNE visualizations of the models’ final-layer features and interpret the clustering as evidence that contrastive learning captures common sentiment features.Source: Image Text CLMLF BERT · pp. 7–8, Figure 6 · p. 8

5.1 Multimodal Sentiment Analysis

• Prior multimodal sentiment methods range from concatenating text and image representations to modeling their interactions with co-memory, attention, fusion, and graph networks.MultiSentiNet and HSAN (Xu, 2017) concatenate encoded representations; CoMN, Yu et al. (2019), MVAN, and Yang et al. (2021) use interaction- or fusion-focused approaches.Source: §5.1 · p. 8

5.2 Contrastive Learning

• CLMLF distinguishes its focus on aligning and fusing token-level features and learning sentiment-related common features from prior contrastive-learning work.Earlier multimodal applications use intramodal, inter-modal, and cross-lingual contrastive learning for video search, or combine within-modality data properties with cross-modal semantics to improve visual representations.Source: §5.2 · p. 8

6 Conclusion and Future Work

• The multi-layer fusion module fuses text and image features at fine-grained token level, supporting local feature fusion.The conclusion contrasts this with previous works’ fusion approaches.Source: §6 · p. 8
• Contrastive learning tasks help CLMLF learn sentiment-related multimodal features and improve its ability to extract and fuse them.Source: §6 · p. 9
• Experiments on public datasets show CLMLF is competitive with strong baselines, while visualizations offer intuitive interpretations of its contrastive tasks and fusion module.Source: §6 · p. 9
• Future work will incorporate other modalities, such as audio, into sentiment detection.Source: §6 · p. 9

• HFM is a binary-classification dataset, while MVSA-Single and MVSA-Multiple are three-class datasets; Table 4 lists data counts by sentiment category.Source: A Dataset Statistics · p. 12, Table 4 · p. 12

B.1 Experimental Setup

• CLMLF’s text input changes to “[CLS] sentence [SEP] aspect [SEP]” for aspect-based sentiment analysis, while image input remains unchanged.The paper reports that this simple modification lets CLMLF work on aspect-based multimodal sentiment tasks.Source: §B.1 · p. 12
• Experiments use Multi-ZOL8, Twitter-15, and Twitter-179, where each sentence has a corresponding aspect attribute.The dataset statistics are reported in Table 5.Source: §B.1 · p. 12
• For Multi-ZOL, which contains multiple images per example, the experiments randomly select one image for fusion and report improvement over text-only input.The passage notes that this selection loses some features but provides no quantitative comparison.Source: §B.1 · p. 12

B.2 Results

• The comparison includes LSTM, AE-LSTM, RAM, MIMN, TomBERT (Yu and Jiang, 2019), and ESAFN (Yu et al., 2019).The listed baselines span sentence-level and attention-based LSTM models, memory-based approaches, and multimodal models.Source: §B.2 · p. 12
• CLMLF achieves good results in the reported comparison, evaluated with Macro-F1 and ACC.The supplied passages do not provide the table's numerical values or identify a winning baseline.Source: §B.2 · pp. 12–13, Table 6 · p. 13
• Ablations support the effectiveness of multi-layer fusion, label-based contrastive learning, and data-based contrastive learning.The ablation results are presented in Table 7.Source: Table 7 · p. 13, §B.2 · p. 13