PaperlayerView exact revision on arXiv

Source-linked AI summary

Vision-Language Pre-Training for Multimodal Aspect-Based Sentiment Analysis

Yan Ling, Jianfei Yu, Rui Xia

arXiv:2204.07955v2cs.CVcs.CLcs.MM

TL;DR

Separate visual and textual pretraining can miss cross-modal alignment, and general vision-language tasks are inadequate for fine-grained MABSA. VLP-MABSA combines a unified multimodal encoder-decoder with task-specific pretraining. It generally outperforms state-of-the-art methods across three MABSA subtasks.

  • Problem

    Separate visual and textual pretraining misses cross-modal alignment, while general vision-language pretraining is inadequate for fine-grained aspects, opinions, and their alignments.

    Source: §1 · pp. 1–2

  • Method

    VLP-MABSA uses a unified multimodal encoder-decoder for pretraining and downstream tasks, with task-specific objectives across language, vision, and multimodal inputs.

    Source: §1 · p. 2

  • Results

    Across three MABSA subtasks, VLP-MABSA generally outperforms state-of-the-art methods; on TWITTER-2015 JMASA, it exceeds JML by 2.5 absolute F1 points.

    Source: §4.3 · p. 7, §5 · p. 9

  • Takeaways & Limitations

    Analysis in supervised and weakly supervised settings supports the effectiveness of each pretraining task.

    Source: §1 · p. 2

    Future work is to apply the approach to a larger dataset and consider image-text relations in pretraining.

    Source: §5 · p. 9

Abstract

from arXiv · show

As an important task in sentiment analysis, Multimodal Aspect-Based Sentiment Analysis (MABSA) has attracted increasing attention in recent years. However, previous approaches either (i) use separately pre-trained visual and textual models, which ignore the crossmodal alignment or (ii) use vision-language models pre-trained with general pre-training tasks, which are inadequate to identify finegrained aspects, opinions, and their alignments across modalities. To tackle these limitations, we propose a task-specific Vision-Language Pre-training framework for MABSA (VLPMABSA), which is a unified multimodal encoder-decoder architecture for all the pretraining and downstream tasks. We further design three types of task-specific pre-training tasks from the language, vision, and multimodal modalities, respectively. Experimental results show that our approach generally outperforms the state-of-the-art approaches on three MABSA subtasks. Further analysis demonstrates the effectiveness of each pretraining task. The source code is publicly released at https://github.com/NUSTM/VLP-MABSA.

1 Introduction

MABSA targets aspect terms and their sentiments in text-image pairs, but prior approaches either miss cross-modal alignment or use general vision-language pretraining inadequate for fine-grained aspect-opinion alignment. VLP-MABSA uses a unified multimodal encoder-decoder and task-specific pretraining, and the paper reports gains across three subtasks.

Task context

  • MATE extracts aspect terms from text-image pairs, MASC classifies sentiment toward each aspect, and JMASA jointly extracts aspect-sentiment pairs.MABSA is also known as Target-Oriented Multimodal Sentiment Analysis or Entity-Based Multimodal Sentiment Analysis.Source: §1 · p. 1

Motivation

  • Separate visual and textual pretraining misses cross-modal alignment, while general vision-language tasks are inadequate for fine-grained aspects, opinions, and their alignments.The paper motivates task-specific vision-language pretraining to address these gaps.Source: §1 · pp. 1–2

Pre-training setup

  • MVSA-Multi supplies pretraining data; representative pretrained models and rule-based methods provide AOE and AOG supervision, while tweet sentiment labels supervise MSP.Source: §1 · p. 2

Framework

  • VLP-MABSA is a unified multimodal encoder-decoder architecture for both pretraining and downstream MABSA tasks.It is based on BART and incorporates aspect, opinion, and sentiment supervision.Source: §1 · p. 2

Pre-training tasks

  • The framework combines Textual Aspect-Opinion Extraction, Visual Aspect-Opinion Generation, and Multimodal Sentiment Prediction with MLM and MRM.The task-specific objectives address fine-grained aspects, opinions, and cross-modal alignments.Source: §1 · p. 2

Results

  • VLP-MABSA generally outperforms state-of-the-art methods on three MABSA subtasks, and analysis supports each pretraining task's effectiveness.The analysis covers supervised and weakly supervised settings.Source: §1 · p. 2

2 Related Work

Related work spans text-based joint aspect-sentiment analysis, coarse-grained multimodal sentiment analysis, and fine-grained MABSA. The paper situates its approach among general vision-language pretraining and task-specific pretraining methods.

Vision-Language Pre-training

  • General vision-language pretraining uses MLM, MRM, and image-text matching, while researchers also develop task-specific pretraining for downstream tasks.This work adds MABSA-specific objectives alongside general pretraining tasks.Source: §2 · p. 2

Text-based Joint Aspect-Sentiment Analysis

  • Text-based JASA extracts aspect terms and predicts sentiment polarities, with pipeline, multitask, collapsed-label, and unified generative approaches described.Yan et al. (2021) proposed a unified generative framework with competitive benchmark performance.Source: §2 · p. 2

Multimodal Sentiment Analysis

  • Coarse-grained multimodal sentiment analysis detects each social post's overall sentiment, whereas this work targets sentiments toward individual aspects.Source: §2 · pp. 2–3

Multimodal Aspect-Based Sentiment Analysis

  • Prior MABSA work addresses aspect term extraction, aspect sentiment classification, and joint aspect-sentiment analysis; this paper aims to pretrain for all three subtasks.Source: §2 · p. 3

3 Methodology

The methodology builds a multimodal BART-based encoder-decoder for vision-language pretraining and downstream MABSA tasks.

3 Methodology

  • BART, a denoising autoencoder for sequence-to-sequence models, is extended to encode textual and visual inputs and decode tasks from different modalities.Source: §3 · p. 3
  • The methodology covers feature extraction, the encoder and decoder, three types of pretraining tasks, and downstream MABSA tasks.Source: §3 · p. 3

3.1 Feature Extractor

The feature extractor represents images as projected region features and text as token embeddings.

Image Representation

  • Faster R-CNN proposes image regions, retains the 36 highest-confidence regions, and supplies mean-pooled convolutional features projected to d-dimensional vectors.The region class distributions are retained for Masked Region Modeling.Source: §3.1 · p. 3

Text Representation

  • Text representation tokenizes the input and uses the token embeddings as text features.Source: §3.1 · p. 3

3.2 BART-based Generative Framework

The framework uses a BART-based generative model for both vision-language pre-training and downstream MABSA tasks. A bidirectional encoder processes text and visual inputs, while a task-conditioned unidirectional decoder generates outputs.

3.2 BART-based Generative Framework

  • A BART-based generative framework serves both vision-language pre-training and downstream MABSA tasks.The encoder combines textual and visual inputs, and the decoder generates task outputs.Source: §3.2 · p. 3
  • The bidirectional Transformer encoder marks visual and textual inputs with separate boundary tokens before concatenating them as the multimodal input.Visual features use ⟨img⟩ and ⟨/img⟩; text uses ⟨bos⟩ and ⟨eos⟩.Source: §3.2 · p. 3
  • The unidirectional decoder shares task-specific generation tokens across the five pre-training tasks.Each task is signaled by ⟨bos⟩ followed by its own token: ⟨mlm⟩, ⟨aoe⟩, ⟨mrm⟩, ⟨aog⟩, or ⟨msp⟩.Source: §3.2 · p. 4

3.3 Pre-training Tasks

Pre-training uses MVSA-Multi and combines textual, visual, and multimodal objectives in a shared framework. The tasks target language recovery and aspect-opinion extraction, visual region and aspect-opinion understanding, and sentiment prediction from both modalities.

3.3 Pre-training Tasks

  • MVSA-Multi supplies image-text pairs and coarse-grained sentiment labels for three types of pre-training: textual, visual, and multimodal.The five objectives are optimized iteratively with a weighted full-pretraining loss.Source: §3.3 · p. 4, §3.3.4 · p. 6

3.3.1 Textual Pre-training

  • MLM masks 15% of text tokens and reconstructs the original text from the masked text and image.This general language task is paired with the task-specific Textual Aspect-Opinion Extraction objective.Source: §3.3.1 · p. 4
  • AOE extracts text aspects and opinions using NER-derived entities and lexicon-matched opinion words, then generates their start and end indexes.The extracted terms provide supervision because MVSA-Multi lacks aspect and opinion annotations.Source: §3.3.1 · p. 4

3.3.2 Visual Pre-training

  • MRM masks 15% of image regions and predicts each masked region’s semantic class distribution against Faster R-CNN detections.Its loss minimizes KL divergence between the predicted and detected distributions.Source: §3.3.2 · p. 5
  • AOG generates image-derived adjective-noun pairs as visual aspect-opinion supervision, using the highest-probability ANP from DeepSentiBank’s 2,089 classes.ANP nouns and adjectives represent visual aspects and opinions, respectively.Source: §3.3.2 · p. 5

3.3.3 Multimodal Pre-training

  • MSP predicts sentiment from multimodal input, using MVSA-Multi’s coarse-grained text-image sentiment labels as supervision.The paper motivates this objective as capturing subjective information across language and vision and their alignments.Source: §3.3.3 · p. 5

3.3.4 Full Pre-training Loss

  • The full pre-training objective uses alternating optimization to iteratively optimize all five tasks, with five trade-off weights controlling their contributions.The weights are λ1 through λ5.Source: §3.3.4 · p. 6

3.4 Downstream Tasks

The downstream tasks are JMASA, MATE, and MASC, all modeled with the same BART-based generative framework used for vision-language pre-training. Their outputs encode aspect spans and, for sentiment tasks, the corresponding sentiment labels.

3.4 Downstream Tasks

  • JMASA, MATE, and MASC use the same BART-based generative framework as pre-training so downstream tasks can benefit during fine-tuning.JMASA jointly extracts aspect terms and their sentiments; MATE extracts terms, and MASC classifies sentiment for aspects.Source: §3.4 · p. 6
  • The tasks generate aspect start and end indexes, with sentiment tokens included for JMASA and MASC.The output indexes mark aspect boundaries; sentiment tokens encode the aspect’s polarity.Source: §3.4 · p. 6
  • For JMASA, the example target sequence encodes Sergio Ramos as positive and UCL as neutral.The sequence uses aspect-span indexes followed by ⟨POS⟩ or ⟨NEU⟩ sentiment markers.Source: §3.4 · p. 6

4 Experiment

Experiments across three MABSA subtasks show broad gains for VLP-MABSA, with analyses examining the pre-training tasks, low-resource performance, and example predictions.

4.1 Settings

  • The evaluation uses TWITTER-2015 and TWITTER-2017 and reports Micro-F1, precision, and recall, plus accuracy for MASC.Experiments use BART-base; the MASC inference setup provides all gold aspects to the decoder.Source: §4.1 · pp. 6–7

4.2 Compared Systems

  • Compared systems span multimodal MATE and MASC methods, text-based JASA models, and multimodal JMASA pipelines, collapsed-label models, and JML.JML includes an auxiliary cross-modal relation-detection task.Source: §4.2 · p. 7

4.3 Main Results

  • VLP-MABSA outperforms JML on JMASA by 2.5 F1 points on TWITTER-2015 and 2.0 on TWITTER-2017, and generally leads on MATE and MASC except TWITTER-2015 accuracy.The JMASA comparison names JML as the second-best system; MATE and MASC results are described as generally best across both datasets.Source: §4.3 · pp. 7–8, Table 4 · p. 7, Table 5 · p. 7

4.4 In-depth Analysis of Pre-training Tasks

  • Adding pre-training tasks generally improves most metrics; in weak supervision on TWITTER-2015, AOE raises MATE F1 by 9.44%, while AOG moderately improves all three subtasks and MSP especially boosts MASC.The analysis also reports only slight improvements from MLM alone and after adding MRM; the full model generally performs best across most subtasks in full and weak supervision.Source: §4.4 · p. 8
  • Pre-training yields a large JMASA improvement with few downstream training samples, but a smaller improvement as the sample size grows.The figure plots JMASA F1 against the number of downstream training samples.Source: §4.4 · p. 8, Figure 3 · p. 8

4.5 Case Study

  • In four test examples, full-pretraining VLP correctly extracts all aspect terms and classifies their sentiment, unlike BART and multimodal inputs without pre-training.The compared systems are BART, the multimodal framework without pre-training (MM), and the fully pre-trained multimodal framework (VLP).Source: §4.5 · pp. 8–9, Table 8 · p. 9

5 Conclusion

The paper concludes that VLP-MABSA combines task-specific pre-training with a unified framework and generally outperforms state-of-the-art methods across three MABSA subtasks. It describes the work as a first step and identifies larger pre-training data and image-text relations as future directions.

5 Conclusion

  • VLP-MABSA is a task-specific vision-language pre-training framework for MABSA, with pre-training tasks spanning language, vision, and multimodal modalities.Source: §5 · p. 9
  • The approach generally outperforms state-of-the-art methods across three MABSA subtasks.Source: §5 · p. 9
  • The authors call this a first step and plan to apply the approach to a larger dataset and consider image-text relations in pre-training.Source: §5 · p. 9
Loading 2204.07955v2…