PaperlayerView exact revision on arXiv

Source-linked AI summary

Exploiting BERT For Multimodal Target Sentiment Classification Through Input Space Translation

Zaid Khan, Yun Fu

arXiv:2108.01682v2cs.CLcs.CV

TL;DR

Multimodal target sentiment classification must identify sentiment toward a target while handling short tweets and complex, potentially irrelevant images. The paper translates images into text and feeds that representation to BERT through an auxiliary sentence, achieving state-of-the-art performance on two Twitter datasets without modifying BERT and explaining a failure mode of BERT-Pair-QA on tweets.

  • Problem

    Short multimodal tweets combine target-dependent sentiment with complex visual content that may be irrelevant or noisy, challenging existing text and multimodal sentiment approaches.

  • Method

    An object-aware transformer performs input-space, non-autoregressive image-to-text translation, and the generated text enters BERT through an auxiliary sentence.

  • Results

    The method achieves state-of-the-art target/aspect sentiment performance on the Twitter-15/17 datasets without modifying the pretrained BERT architecture.

  • Takeaways & Limitations

    Input translation preserves object-level information from complex images while enabling multimodal fusion through an unchanged language model.

Abstract

from arXiv · show

Multimodal target/aspect sentiment classification combines multimodal sentiment analysis and aspect/target sentiment classification. The goal of the task is to combine vision and language to understand the sentiment towards a target entity in a sentence. Twitter is an ideal setting for the task because it is inherently multimodal, highly emotional, and affects real world events. However, multimodal tweets are short and accompanied by complex, possibly irrelevant images. We introduce a two-stream model that translates images in input space using an object-aware transformer followed by a single-pass non-autoregressive text generation approach. We then leverage the translation to construct an auxiliary sentence that provides multimodal information to a language model. Our approach increases the amount of text available to the language model and distills the object-level information in complex images. We achieve state-of-the-art performance on two multimodal Twitter datasets without modifying the internals of the language model to accept multimodal data, demonstrating the effectiveness of our translation. In addition, we explain a failure mode of a popular approach for aspect sentiment analysis when applied to tweets. Our code is available at \textcolor{blue}{\url{https://github.com/codezakh/exploiting-BERT-thru-translation}}.

1 INTRODUCTION

Multimodal target sentiment analysis addresses the challenge of identifying sentiment toward entities in short, emotionally charged tweets whose accompanying images may be noisy or complex. The proposed approach translates images into text and uses an auxiliary sentence to provide multimodal information to BERT without changing its architecture.

  • 1 INTRODUCTION: Multimodal target sentiment analysis combines target-oriented sentiment understanding with vision-language modeling for emotionally rich social-media content.Twitter provides a critical setting because emotional narratives and misinformation spread there, while target sentiment requires identifying what the sentiment is directed toward.
  • 1 INTRODUCTION: Short tweets, potentially irrelevant images, and complex visual scenes make direct transfer from text-only or conventional multimodal methods challenging.These conditions differ from review texts and speaker-focused videos commonly used in related fields.
  • 1 INTRODUCTION: The model translates images into text with an object-aware transformer, then feeds the translation to BERT through an auxiliary sentence.This input-space translation increases the text available to the language model while preserving complex scene information in natural language.
  • 1 INTRODUCTION: The approach draws on DETR-style transformer layers but adapts them for non-autoregressive text generation rather than object detection.The contribution centers on adapting an object-detection architecture to generate textual image representations.
  • 1 INTRODUCTION: The auxiliary sentence provides a way to fuse multimodal information through a large pretrained language model without modifying the model’s internals.The architecture translates in input space rather than feature space and does not require a GAN.

2 RELATED WORK

Related work spans overall multimodal sentiment analysis, target/aspect sentiment classification, vision-language transformers, and multimodal target-oriented sentiment classification. The paper positions its approach as using image-to-text translation and auxiliary-sentence fusion while retaining the baseline BERT architecture.

  • 2.1 Multimodal Sentiment Classification: Classical multimodal sentiment analysis classifies overall utterance sentiment from acoustic, visual, and language cues, typically using speaker-focused videos where the visual modality is relevant.Social-media images can instead be irrelevant or lack emotional content, and the accompanying modality is a single image rather than a sequence.
  • 2.2 Aspect Sentiment Classification: Target/aspect sentiment classification determines sentiment toward a target mentioned in text, with pretrained language models such as BERT dominating the field.The paper notes that architectural modifications to large language models can be brittle and may not transfer across implementations and applications.
  • 2.2 Aspect Sentiment Classification: BERT-Pair converts multiclass aspect sentiment classification into binary sentence-pair problems using an auxiliary sentence that encodes label information as a query.The proposed work instead uses its auxiliary sentence to pass multimodal information into BERT.
  • 2.3 Transformers for Vision-Language Tasks: Vision-language transformers process language and visual inputs in separate streams before fusing them, and the paper’s caption transformer is inspired by DETR.The proposed translation maps visual information into the language-model input space rather than requiring native multimodal internals.
  • 2.4 Multimodal Target Oriented Sentiment Classification: Multimodal target-oriented sentiment classification combines multimodal sentiment analysis with target-oriented sentiment classification and is commonly studied on online social-media data.Compared with TomBERT, the proposed approach does not modify baseline BERT beyond one linear layer or add self-attention layers.

3 FUSION AND SENTIMENT ANALYSIS THROUGH INPUT TRANSLATION

EF-CaTrBERT translates images into symbolic natural-language input, then combines that description with the target in an auxiliary sentence for BERT-based sentiment prediction.

  • 3.1 Problem Definition: The task predicts negative, neutral, or positive sentiment toward an opinion target using the target, sentence, and accompanying image.
  • 3.2 Overview: The captioning transformer converts an image tensor into a fixed-length vector of vocabulary-token integers representing a natural-language description.Shared tokenizers and vocabulary mapping place image context in the language model’s symbolic input space.
  • 3.3.1 Transformer Encoder: A CNN backbone reduces image features before DETR encoder layers process spatially flattened representations with positional encodings and multi-head attention.The backbone uses ResNet 101, reduces 2048 channels to d=256, and flattens the spatial dimensions.
  • 3.3.2 Transformer Decoder: Unlike DETR’s object-query decoder, the model predicts a complete image description non-autoregressively in one forward pass.Each position produces a probability distribution over BERT’s 30522-token vocabulary using learned projections and ReLU activations.
  • 3.4 Fusion Through The Auxillary Sentence: The model concatenates target tokens with predicted image-description tokens as Sentence B, pairs it with the tweet as Sentence A, and uses BERT’s [CLS] pooler output for classification.A learned linear layer maps the 768-dimensional [CLS] representation to negative, neutral, and positive sentiment probabilities.

4 EXPERIMENTS

The experiments test whether input-space translation improves text-only language models, supports multimodal correspondence, scales with multimodal information, and reveals unimodal failure modes.

  • Research Questions: The study evaluates input-space translation against purely text-based, multimodal, and unimodal approaches.
  • Research Questions: The experiments examine whether generated image descriptions help BERT relate images to tweets and targets to images.
  • Research Questions: The study also evaluates how the amount of multimodal information affects prediction accuracy.
  • Research Questions: A final research question investigates failure modes of state-of-the-art unimodal models on short-text social media data.

4.1 Experiment Settings

Experiments use benchmark multimodal Twitter datasets for target-oriented sentiment classification, an image–tweet relationship dataset, and MS-COCO for caption-transformer training.

  • Datasets: Twitter-15 and Twitter-17 contain tweets, accompanying images, targets, and target-specific negative, neutral, or positive labels.Both datasets define a standard multiclass target-oriented multimodal sentiment classification task.
  • Datasets: The Bloomberg Twitter-Image Relationship dataset classifies whether tweet entities are represented in the accompanying image.Its binary labels are TEXT IS REPRESENTED and TEXT NOT REPRESENTED.
  • Datasets: MS-COCO trains the caption transformer, without a separate image-captioning evaluation because image captioning is not the study’s goal.

4.2 Multimodal Target Oriented Sentiment Analysis Performance (RQ1)

EF-CapTrBERT and EF-CapTrBERT-DE outperform text-only and multimodal baselines, while auxiliary-sentence fusion is central to their advantage.

  • Ablation: LF-CapTrBERT performs much worse than the EF configurations because it encodes the tweet and caption separately instead of using the auxiliary sentence.This comparison serves as an ablation of the auxiliary-sentence mechanism.
  • Stability: EF-CapTrBERT-DE has lower stability than EF-CapTrBERT, which the authors conjecture results from domain mismatch between the caption transformer and language model.The caption transformer remains unchanged while BERT receives domain-specific initialization.

4.3 Image-Relationship Understanding (RQ2)

Adding image context enables the model to improve over text-only BERT when determining relationships among the image, target, and tweet.

  • Results: Adding multimodal image context with EF-CapTrBERT substantially improves performance over text-only BERT on the Bloomberg Twitter-Image Relationship Dataset.Accuracy and weighted-F1 are reported for comparison with prior work.
  • Interpretation: EF-CapTrBERT learns when to seek additional image context to understand the relationship between the image, target, and tweet.The authors attribute this capability to sufficiently effective fusion of image and text.

4.4 Further Analysis RQ3 + RQ4

Longer image descriptions do not reliably improve accuracy because added information can bring noise, while BERT-Pair-QA degrades through overconfident predictions on tweets.

  • RQ3: Description Length: There is no strong relationship between image-caption descriptiveness and prediction accuracy.Caption length is used as a weak proxy for the amount of information in the description.
  • RQ3: Description Length: Longer captions may add noise and spurious objects that counteract their potential informational benefit.Thus, additional caption content is not necessarily useful multimodal information.
  • RQ4: BERT-Pair-QA: BERT-Pair-QA converts three-class sentiment classification into three binary classifiers and selects a class using their confidence levels.This differs from the proposed method’s direct multimodal use of an auxiliary sentence.
  • RQ4: BERT-Pair-QA: BERT-Pair-QA consistently overestimates probabilities on tweets, which the authors identify as the reason for its performance degradation.The method was developed for longer, more informative review texts, where confidence levels may be more trustworthy.

5 CONCLUSION

The paper addresses short texts, variable visual usefulness, and complex scenes with input-space image translation and auxiliary-sentence fusion, achieving strong multimodal sentiment results without modifying BERT.

  • Conclusion: Multimodal target sentiment analysis combines short texts, variably useful visual modalities, and complex scenes that distinguish it from related settings.These challenges motivate a specialized approach for multimodal tweets.
  • Conclusion: EF-CaTrBERT translates images into text in input space and passes the result into a language model through an auxiliary sentence.The approach performs multimodal fusion without requiring multimodal modifications to the language-model architecture.
  • Conclusion: The auxiliary-sentence configuration outperforms the corresponding late-fusion configuration, supporting the role of auxiliary-sentence fusion.The conclusion also reports improved entity-image relationship detection and state-of-the-art Twitter-15/17 sentiment performance.
  • Conclusion: The paper explains why BERT-Pair-QA, despite being powerful elsewhere, suffers performance degradation on tweets.The analysis attributes this degradation to the model’s confidence behavior in the social-media setting.
Loading 2108.01682v2…