Source-linked AI summary
A Multi-task Approach for Named Entity Recognition in Social Media Data
Gustavo Aguilar, Suraj Maharjan, Adrian Pastor López-Monroy, Thamar Solorio
TL;DR
Noisy social-media text makes named entity recognition difficult because entities are sparse and lack strict rules. The paper jointly learns fine-grained entity categories and entity segmentation, then uses a CRF classifier, achieving 41.86% entity F1 and 40.24% surface F1 on WNUT-2017.
Problem
Social-media named entity recognition is difficult because entities are sparse in limited annotated data and do not follow strict rules.
Method
A multi-task neural network jointly predicts entity categories and entity segmentation using character, word, POS, and gazetteer features before CRF classification.
Results
41.86% entity F1-score and 40.24% surface F1-score were achieved on the WNUT-2017 shared task.
Takeaways & Limitations
The CRF consistently improves neural-network results, while POS tags and gazetteers contribute to performance.
Takeaways & Limitations
Error analysis identifies entity-boundary errors and frequent disregard of fully capitalized entities.
Abstract
from arXiv · showhide
Named Entity Recognition for social media data is challenging because of its inherent noisiness. In addition to improper grammatical structures, it contains spelling inconsistencies and numerous informal abbreviations. We propose a novel multi-task approach by employing a more general secondary task of Named Entity (NE) segmentation together with the primary task of fine-grained NE categorization. The multi-task neural network architecture learns higher order feature representations from word and character sequences along with basic Part-of-Speech tags and gazetteer information. This neural network acts as a feature extractor to feed a Conditional Random Fields classifier. We were able to obtain the first position in the 3rd Workshop on Noisy User-generated Text (WNUT-2017) with a 41.86% entity F1-score and a 40.24% surface F1-score.
1 Introduction
Social-media NER is difficult because entities are sparse, irregular, and embedded in noisy user-generated text. The paper addresses this with joint NE segmentation and fine-grained categorization, achieving top shared-task results.
- Social-media NER is challenging because entities occupy a small portion of limited annotated data and do not follow strict rules.These properties make generalization difficult.
- The proposed approach jointly trains fine-grained NE categorization with auxiliary binary NE segmentation.Segmentation predicts whether each token is an NE or not.
- 41.86% entity F1-score and 40.24% surface F1-score were the best results among WNUT-2017 shared-task participants.The evaluation used the shared-task data.
2 Related Work
Prior NER work progressed from hand-crafted features toward neural architectures, while multitask learning offered a way to improve generality across related tasks. This paper applies that setting by adding NE segmentation to NE categorization.
- Traditional NER systems rely on hand-crafted features and external resources, whereas neural systems can achieve strong performance without them.Hand-crafted features are described as expensive to develop and maintain.
- This paper distinguishes itself by combining CNN and BLSTM representations in parallel and adding POS tags and gazetteer representations.Its main contribution is framed as multitask learning plus these feature representations.
- Multitask learning has been used to improve generality across shared NLP tasks.Prior work connected multitask success with factors including label entropy, regularization, and training-data size.
3 Methodology
The methodology combines character, word, POS, and gazetteer representations in a multitask neural network. Its learned features are then supplied to a CRF for sequence-level label prediction.
- Feature Representation: Character features encode capitalization, punctuation, word shape, and other orthographic patterns, including non-ASCII handling.The encoded character sequence is processed by two stacked convolutional layers and global average pooling.
- Feature Representation: Gazetteer features are six-dimensional binary vectors indicating whether a word appears in the gazetteer for each entity class.The gazetteers are intended to help precision for well-known entities.
- Model Description: Word-level representations combine pretrained Twitter word embeddings with POS-tag embeddings processed by a BLSTM.The BLSTM is used to capture contextual information from word and POS sequences.
- Model Description: The multitask output uses a sigmoid single-neuron segmentation layer and a softmax 13-neuron categorization layer.The two task losses are added and backpropagated jointly.
- Model Description: The system jointly predicts NE segmentation and categorization, then uses the neural network weights as features for a CRF classifier.The CRF predicts the most likely label sequence rather than making independent token-level decisions.
4 Experimental Settings
The experiments preprocess URLs uniformly, split development data between validation and evaluation, and use specified neural-network and CRF configurations. The setup compares the multitask network with its CRF-based sequential inference.
- URLs are replaced with <URL>, and the development set is divided equally between validation and evaluation.
- Figure 3 organizes the system around CNN, BLSTM, and dense encoders whose concatenated features support segmentation, categorization, and CRF inference.
- The CNN uses kernel size 3 and 64 filters in both convolutional layers, while the BLSTM uses dropout rates of 0.5 before and after its layers.Increasing CNN filters or convolutional layers worsened results and increased computation time.
- The CRF is trained with L-BFGS and L1 and L2 regularization penalties of 1.0 and 1.0e−3, respectively.
5 Results and Discussion
The multi-task network paired with CRF consistently outperformed the neural network alone, while POS tags and gazetteers improved development results. In WNUT-2017, the submission achieved its strongest performance for person and location, but remained challenged by creative-work and product entities.
- 5 Results and Discussion: The CRF consistently outperformed the multi-task neural network alone across experiments, highlighting the value of sequential information.Table 1 reports CRF classification results on the development set used for both validation and evaluation.
- 5 Results and Discussion: 54.12% F1-score was achieved when the multi-task network fed features to the CRF, improving over the 52.42% single-task result.The authors report that adding the secondary NE-segmentation task improved the CRF result.
- 5 Results and Discussion: Removing gazetteers reduced F1-score from 54.12% to 52.69%, while removing POS tags reduced it to 51.12%.The complete feature set produced the best result among the tested combinations.
- 5 Results and Discussion: The final system performed best for person and location, whereas creative-work was the hardest category and product was also difficult.The authors attribute the latter difficulty to rapidly growing entities with less restrictive patterns.
6 Error Analysis
Error analysis identifies named-entity boundary detection and capitalization-related domain shifts as recurring failure patterns.
- 6 Error Analysis: The CRF often misidentified entity boundaries by absorbing preceding articles or following conjunctions into named entities.It also tended to merge adjacent entities when a conjunction connected them.
- 6 Error Analysis: Fully capitalized named entities were frequently disregarded, possibly because capitalization patterns differ across Twitter, Reddit, and Stack Overflow.Twitter abbreviations may be capitalized without being entities, while capitalized words in the other domains are more likely to denote entities.
7 Conclusion
The approach extracts relevant features from noisy user-generated text and combines multi-task learning with CRF classification. The authors highlight POS tags, gazetteers, and embeddings as relevant, while noting that further work is needed to understand model strengths and weaknesses.
- A CRF classifier boosts neural-network results by using the whole sentence to predict the most likely label set.This complements the feature extraction performed by the multi-task neural network.
- POS tags and gazetteers are important representations for the NER task.
- Twitter word embeddings and orthographic character embeddings are relevant to the task.
- The authors plan to improve results by better understanding the model’s strengths and weaknesses and evaluating it on related noisy-NE tasks.