Source-linked AI summary
RpBERT: A Text-image Relation Propagation-based BERT Model for Multimodal NER
Lin Sun, Jiquan Wang, Kai Zhang, Yindu Su, Fangsheng Weng
TL;DR
Multimodal NER often uses visual clues without determining whether the image is relevant to the tweet, so unrelated images can introduce harmful attention. RpBERT propagates inferred text-image relations through soft or hard gates and trains relation classification with MNER. It reports best F1 scores on TRC and both MNER datasets.
Problem
Existing multimodal NER methods extract or concatenate visual features regardless of text-image relevance, allowing unrelated images to interfere with linguistic inference.
Method
RpBERT extends multimodal BERT with text-image relation propagation, soft or hard probability gates, and multitask training for relation classification and MNER.
Results
RpBERT achieves the best reported F1 scores of 88.1% on TRC, 74.9% on Fudan Univ., and 87.8% on Snap Res.
Takeaways & Limitations
Relation propagation adaptively leverages visual information according to text-image relations and resolves failure cases reported for earlier multimodal models.
Abstract
from arXiv · showhide
Recently multimodal named entity recognition (MNER) has utilized images to improve the accuracy of NER in tweets. However, most of the multimodal methods use attention mechanisms to extract visual clues regardless of whether the text and image are relevant. Practically, the irrelevant text-image pairs account for a large proportion in tweets. The visual clues that are unrelated to the texts will exert uncertain or even negative effects on multimodal model learning. In this paper, we introduce a method of text-image relation propagation into the multimodal BERT model. We integrate soft or hard gates to select visual clues and propose a multitask algorithm to train on the MNER datasets. In the experiments, we deeply analyze the changes in visual attention before and after the use of text-image relation propagation. Our model achieves state-of-the-art performance on the MNER datasets.
Introduction
Multimodal NER can be harmed when attention extracts visual clues from text-image pairs that are unrelated. RpBERT propagates inferred text-image relations to regulate visual information and reports state-of-the-art MNER performance.
- Motivation: Approximately 56% of Vempala’s TRC instances and 60% of Twitter100k pairs were classified as having images that do not add to tweet meaning.These proportions indicate that irrelevant text-image pairs are common in the examined data.
- Motivation: Attention-based models can focus on irrelevant regions, such as a wall and ground, causing “Cleveland” to receive the wrong LOC label.The cited failure case concerns an entity labeled ORG in the text.
- Approach: RpBERT introduces text-image relation propagation with soft and hard propagation mechanisms in a multimodal BERT model.The model is designed to address inappropriate visual attention clues.
- Approach: The model trains text-image relation classification and downstream NER as multiple tasks.The multitask procedure connects relation inference with MNER training.
- Findings: Relation propagation reduces interference from irrelevant images and leverages more visual information for relevant text-image pairs.The paper also analyzes visual attention using numerical distributions and heat maps.
- Findings: The experimental results report correctly recognized failure cases and state-of-the-art performance on MNER datasets.The introduction states both qualitative correction of failures and a best-performing result.
Related Work
Earlier multimodal NER and visual-linguistic models incorporate visual features through modality combination, attention, or cross-modal transformers. The paper highlights that these approaches may use salient visual features even when images are irrelevant to the text.
- Multimodal NER: Earlier MNER models combine word, character, and visual embeddings or use visual attention to incorporate image regions into token-level representations.Moon et al. use modality attention, while Lu et al. use text-conditioned visual attention.
- Visual-Linguistic Pretraining: Visual-linguistic pretrained models use cross-modality or coattentional transformer layers to connect textual and visual streams.Their visual representations commonly use RoI or block-region features.
- Limitations: For irrelevant text-image pairs, salient visual features can interfere with linguistic features, while object categories may not cover entities such as company trademarks and scenic locations.The passage also describes pretraining with masked language, masked region, and image-text matching tasks.
- Motivation: Existing multimodal models concatenate or attention-extract visual features regardless of whether images contribute to text semantics, motivating RpBERT for varied text-image relations.The paper links this design issue to failed MNER examples.
The Proposed Approach
RpBERT extends BERT into a multitask multimodal NER framework that infers text-image relations and propagates them through soft or hard gates. It masks or weights visual features before NER decoding and trains relation classification and MNER jointly.
- Architecture: RpBERT extends vanilla BERT into a multitask framework for text-image relation classification and visual-linguistic learning for MNER.The architecture uses two RpBERTs with shared structure and parameters.
- Architecture: The input sequence contains [CLS], linguistic tokens, [SEP], and visual features, with [CLS] used for text-image relation classification.T denotes linguistic features and V denotes visual features.
- Visual Representation: ResNet extracts 49 image block regions, which are projected to BERT’s embedding size and combined with segment and position embeddings.The final ResNet convolutional output has size 7 × 7 × d_v, with d_v = 2048 for ResNet-152.
- Relation Inference: A binary classifier produces probabilities [π0, π1], and the positive probability defines the text-image relevant score r used to construct a visual mask matrix R.The mask controls how visual features are propagated into the model.
- Relation Propagation: Soft propagation continuously filters visual features according to text-image relation strength, whereas hard propagation selects or discards them using 0 or 1.The two behaviors are implemented with different probability gates.
- Relation Propagation: For hard propagation, Gumbel-Softmax provides a continuous approximation to categorical gating, becoming one-hot as temperature approaches 0.Training anneals the temperature from 1 to 0.1.
- Multitask Training for MNER: The propagated token outputs are concatenated with word and character embeddings and fed to a biLSTM-CRF for entity tagging.CRF assigns entity labels using the biLSTM hidden vectors.
- Multitask Training for MNER: Each training epoch first optimizes relation classification on TRC data and then optimizes MNER on the MNER dataset.The test stage applies relation propagation and decodes labels with Viterbi.
Experiments
RpBERT is evaluated on one text-image relation classification dataset and two multimodal NER datasets, with comparisons against baselines, pretrained models, and ablations. Results show gains from visual clues and relation propagation, including stronger attention alignment with text-image relevance.
- Datasets: RpBERT is evaluated on the TRC dataset and the Fudan University and Snap Research MNER datasets.The TRC dataset supports text-image relation classification, while the other two datasets evaluate multimodal NER.
- Result of TRC: 7.1%: RpBERT increases test-set F1 over Lu et al. (2018) in text-image relation classification.The comparison is reported on the TRC test set.
- Results of MNER: 4.5% and 7.3%: the best “+ RpBERTGs” improves over biLSTM-CRF on Fudan Univ. and Snap Res., respectively.The reported gains quantify the benefit of the strongest RpBERT visual-clue configuration over the text-only baseline.
- Results of MNER: 1.5% and 2.5%: biLSTM-CRF + RpBERT-LargeGs exceeds current best results on Fudan Univ. and Snap Res., respectively.RpBERTGs also outperforms the compared pretrained visual-linguistic models.
- Case Study via Attention Visualization: 20%: relation propagation reduces mean visual attention from 0.041 to 0.034 on irrelevant pairs, while relevant-pair attention rises from 0.042 to 0.102.The attention distribution therefore varies with the relevant score instead of remaining nearly unrelated to it.
Conclusion
RpBERT addresses text-unrelated visual attention in multimodal learning by propagating text-image relations within a multitask model. The experiments report adaptive visual-information use and best F1 scores across TRC and two MNER datasets.
- RpBERT uses text-image relation inference with multitask training for text-image relation classification and downstream NER.
- The ablation study quantitatively evaluates the role of text-image relation propagation, while heat maps and numerical distributions analyze visual attention.
- Relation propagation adaptively leverages visual information according to the relation between text and image.
- 88.1% on the TRC dataset, 74.9% on the Fudan Univ. dataset, and 87.8% on the Snap Res. dataset are the model’s best F1 scores.