Source-linked AI summary
Multimodal Named Entity Recognition for Short Social Media Posts
Seungwhan Moon, Leonardo Neves, Vitor Carvalho
TL;DR
NER on short, noisy social media posts is hindered by limited textual context, ambiguous entity names, and misspelled or unknown tokens. The paper introduces MNER with the SnapCaptions dataset, visual features, and modality attention that selects informative modalities. The resulting system outperforms traditional text-based NER baselines by combining visual and textual information.
Problem
Short, noisy social media posts provide limited textual context, contain polysemous entities, and include unknown tokens caused by inconsistent lexical notation.
Method
The paper builds an image-and-text NER system with visual embeddings and modality attention that adaptively amplifies informative word, character, or visual features.
Results
The proposed MNER system with modality attention outperforms traditional text-based NER baselines on short social media posts.
Takeaways & Limitations
Visual information and adaptive modality integration provide an effective approach for named entity recognition in multimodal social media posts.
Takeaways & Limitations
Modality attention can amplify misleading character or visual contexts, causing incorrect predictions for tokens such as HUUUGE and Shampooer.
Abstract
from arXiv · showhide
We introduce a new task called Multimodal Named Entity Recognition (MNER) for noisy user-generated data such as tweets or Snapchat captions, which comprise short text with accompanying images. These social media posts often come in inconsistent or incomplete syntax and lexical notations with very limited surrounding textual contexts, bringing significant challenges for NER. To this end, we create a new dataset for MNER called SnapCaptions (Snapchat image-caption pairs submitted to public and crowd-sourced stories with fully annotated named entities). We then build upon the state-of-the-art Bi-LSTM word/character based NER models with 1) a deep image network which incorporates relevant visual context to augment textual information, and 2) a generic modality-attention module which learns to attenuate irrelevant modalities while amplifying the most informative ones to extract contexts from, adaptive to each sample and token. The proposed MNER model with modality attention significantly outperforms the state-of-the-art text-only NER models by successfully leveraging provided visual contexts, opening up potential applications of MNER on myriads of social media platforms.
1 Introduction
Social media posts offer valuable information but make NER difficult because they are short, noisy, and often lack enough textual context. The paper addresses these challenges with visual context and modality attention that adaptively selects among words, characters, and images.
- Social media posts contain information useful for understanding events, opinions, and preferences, but it is largely embedded in unstructured text and images.
- Short social media text often lacks enough context to resolve whether a polysemous term such as “monopoly” is a named entity or an economics term.
- Noisy lexical notation creates unknown tokens, such as the misspelled “Marshmelloooo,” making word-embedding-based NER models vulnerable.
- Visual context can disambiguate entities by adding information unavailable in the text, such as identifying “monopoly” as a board game.
- Modality attention selects the most informative word, character, or visual modality at each decoding step while suppressing irrelevant context.
- The proposed LSTM-CNN hybrid multimodal NER network processes image and text, and outperforms state-of-the-art NER models on the SnapCaptions dataset.
2 Related Work
Prior NER work models word and character information, while related multimodal research combines parallel modalities for tasks such as translation. This paper introduces visual context into NER and presents modality attention as a distinct integration mechanism.
- Neural NER systems commonly combine recurrent sequence models, CRFs, and word- and character-level feature extractors for standard NER tasks.
- Attention research often selects subsets within a single input, while related NLP work had not incorporated visual components for noisy social media data.
- Multimodal learning builds joint models that extract contextual information from multiple modalities in parallel datasets.
- Multimodal machine translation uses a source-language sentence together with a corresponding image as input.
- This paper presents, to the authors’ knowledge, the first work incorporating visual contexts into named entity recognition.
3 Proposed Methods
The proposed MNER architecture represents each token through word, character, and visual modalities, combines them with adaptive modality attention, and predicts labels using Bi-LSTM-CRF. It addresses failures from equally merging modalities by weighting each modality according to its informativeness at each decoding step.
- 3.1 Features: Each token is represented with word, character, and visual embeddings before being processed by the entity Bi-LSTM-CRF.The model uses pre-trained GloVe word embeddings, a character Bi-LSTM, and visual features from a modified Inception network trained on ImageNet.
- 3.1 Features: Visual embeddings are supplied to the NER decoder at every decoding step, together with modality attention, rather than only as an initial decoder input.The authors empirically observe better results with this design.
- 3.3 Modality Attention: Modality attention learns a weighted average of available modalities, producing a single context vector for the entity LSTM.The module maps modalities into a unified representation space and weights their importance at each decoding step.
- 3.3 Modality Attention: The attention mechanism addresses inaccurate naive concatenation by suppressing unreliable word embeddings for unknown tokens and emphasizing character or visual context when more informative.Naive merging treats modalities as equally informative, although images may be irrelevant and unknown word embeddings can degrade performance on social-media data.
- 3.3 Modality Attention: The same modality-attention module can operate with only word and character embeddings when visual context is unavailable, and is flexible across NER architectures or other multimodal applications.The text-only formulation excludes visual embeddings, while the module itself is described as architecture-independent.
4 Empirical Evaluation
The evaluation uses SnapCaptions to compare text-only and multimodal NER configurations, testing entity recognition and segmentation across varying modalities and vocabulary conditions. Results show that visual context and modality attention improve predictions, especially for ambiguous or unknown tokens, while misleading signals still cause errors.
- 4.1 SnapCaptions Dataset: SnapCaptions contains 10K manually annotated image-caption pairs with PER, LOC, ORG, and MISC entity labels, split into train, validation, and test sets.Captions average 30.7 characters and 5.81 words; 6,612 of 15,733 vocabulary items are unknown to Stanford GloVE embeddings.
- 4.2 Baselines: The evaluation reports precision, recall, and F1 for both entity-type recognition and untyped entity segmentation.Compared configurations vary word, character, and visual modalities, including proposed modality-attention models and established baselines.
- 4.3 Results: SnapCaptions Dataset: Adding visual context greatly improves performance over text-only W+C models, while modality attention further improves entity-type recognition for W+C+V.The text-only modality-attention model also significantly outperforms state-of-the-art W+C baselines using the same textual modalities.
- 4.3 Results: SnapCaptions Dataset: Visual context resolves ambiguous tokens such as “curry” by supplying scene-related evidence unavailable from surrounding text alone.Visual tags such as “parade” and “urban area” align with training examples associated with Stephen Curry and support the named-entity prediction.
- 4.3 Results: SnapCaptions Dataset: Modality attention improves robustness to unknown tokens by amplifying character-based contexts and suppressing unreliable word embeddings.Examples include misspelled or unseen tokens such as WaRriOoOrs and Kooler Matic; vocabulary-removal experiments show that attention suppresses performance degradation as missing tokens increase.
- 4.3 Results: SnapCaptions Dataset: Incorrect predictions occur when attention amplifies misleading character or visual signals, such as capitalization or concert-associated imagery.The model incorrectly labels HUUUGE and Shampooer as named entities in these cases.
5 Conclusions
The paper introduces multimodal NER for short social media posts and proposes visual integration and modality attention to improve entity representations.
- Visual information is combined with textual information to outperform traditional text-based NER baselines.
- The modality attention mechanism attenuates irrelevant modal information while amplifying the primary modality for better representations.
- The modality-attention model also outperforms state-of-the-art baselines with text only by combining word- and character-level information.
- The approach can be applied to social media posts and other articles containing text and accompanying images.