PaperlayerView exact revision on arXiv

Source-linked AI summary

TENER: Adapting Transformer Encoder for Named Entity Recognition

Hang Yan, Bocao Deng, Xiaonan Li, Xipeng Qiu

arXiv:1911.04474v3cs.CLcs.LG

TL;DR

The paper addresses Transformer encoders’ weak NER performance despite their broad NLP success and parallelism advantages. TENER adapts Transformer attention with direction- and distance-aware relative positions, un-scaled attention, and character encoding, achieving stronger results across six NER datasets and outperforming prior models under matched resources.

  • Problem

    Transformer encoders model long-range context and parallelize well but perform poorly on NER, where directional and distance information matter.

  • Method

    TENER adapts Transformer encoders with direction-aware, distance-aware relative positional encoding and un-scaled attention, and uses the adapted Transformer as a character encoder.

  • Results

    Under the same pre-trained embeddings and external knowledge, the modifications outperform previous models on six NER datasets.

  • Takeaways & Limitations

    The adapted Transformer is effective for NER and its character encoder performs better than BiLSTM and CNN character encoders on two English NER datasets.

  • Takeaways & Limitations

    The paper includes an author-stated property concerning sinusoidal position embeddings for a fixed offset k and position t.

Abstract

from arXiv · show

The Bidirectional long short-term memory networks (BiLSTM) have been widely used as an encoder in models solving the named entity recognition (NER) task. Recently, the Transformer is broadly adopted in various Natural Language Processing (NLP) tasks owing to its parallelism and advantageous performance. Nevertheless, the performance of the Transformer in NER is not as good as it is in other NLP tasks. In this paper, we propose TENER, a NER architecture adopting adapted Transformer Encoder to model the character-level features and word-level features. By incorporating the direction and relative distance aware attention and the un-scaled attention, we prove the Transformer-like encoder is just as effective for NER as other NLP tasks.

1 Introduction

TENER adapts Transformer encoders for NER because vanilla Transformer attention does not adequately capture the directional and distance information needed for entity labeling. The model combines revised relative positional encoding, un-scaled attention, and Transformer-based character encoding to improve NER performance.

  • Task and motivation: NER identifies entity spans and assigns each entity a class, typically using word embeddings, a context encoder, and a decoder.NER supports downstream tasks including question generation, relation extraction, and coreference resolution.
  • Task and motivation: Transformer offers long-range context modeling and better parallelism than RNNs, but has been reported to perform poorly on NER.BiLSTM encoders remain widely used because they learn contextual word representations effectively.
  • Direction- and distance-aware attention: Vanilla sinusoidal position embeddings encode distance but not direction, although NER depends on both directional context and distances between words.Words before “Inc.” and after “in” provide different entity cues, while entities must form continuous spans.
  • Direction- and distance-aware attention: TENER replaces absolute position encoding with revised relative positional encoding that uses fewer parameters and performs better.The revised encoding is intended to make Transformer attention aware of both direction and distance.
  • Un-scaled attention: TENER abandons the dot-product attention scale factor to produce sharper attention, since sparse attention can focus on the few contextual words needed for labeling.The paper argues that smooth attention may include noisy information.
  • Character-level encoding: The model also applies Transformer as a character encoder, targeting character-level patterns that CNN encoders may miss, including varied n-grams and discontinuous patterns.The paper compares four kinds of character encoders.
  • Results: Across six NER datasets, the adapted Transformer outperforms previous models under the same pre-trained embeddings and external knowledge, including BiLSTM-based models.The paper reports state-of-the-art performance among models without pre-trained language models or designed features.

2 Related Work

NER models have progressed from BiLSTM- and character-encoder-based architectures toward Transformer encoders that model token interactions and positional information. The related work describes Transformer components, positional embeddings, and TENER’s use of Transformer encoding for both word- and character-level information.

  • NER architectures: BiLSTM-CRF became a widely used architecture for NER sequence labeling.
  • Character-level features: Character-level information is commonly extracted with CNNs or BiLSTMs because word shape features such as capitalization and n-grams aid entity recognition.
  • Contextual representations: Contextual word embeddings, including ELMo and BERT, were combined with neural NER models to improve performance.
  • Transformer: Transformer self-attention requires position embeddings because attention alone is unaware of token positions.
  • Transformer encoder: The Transformer encoder projects input representations into query, key, and value spaces, applies multi-head self-attention, and then uses position-wise feedforward networks.
  • TENER: TENER uses Transformer encoders to extract word-level context and encode character-level information within words.

3 Proposed Model

The proposed TENER model adapts Transformer encoders for NER by modeling character- and word-level features and addressing shortcomings in positional attention. Its adaptations use relative direction- and distance-aware attention, unscaled attention, and parameter reductions, with CRF decoding.

  • 3 Proposed Model: TENER uses Transformer encoders to model long-range and complicated sentence interactions for NER, including character-level word representations.The final word embedding concatenates character features with pretrained word embeddings.
  • 3 Proposed Model: The adapted Transformer addresses NER’s weaker performance than BiLSTM models with two principal improvements.The paper analyzes properties that Transformer lacks compared with BiLSTM-based encoders.
  • 3.2.1 Direction- and Distance-Aware Attention: Vanilla Transformer attention can break the distance information in sinusoidal embeddings after projection into query and key spaces.The paper reports that projected positional embeddings have no clear distance pattern, motivating the adapted attention calculation.
  • 3.2.1 Direction- and Distance-Aware Attention: Relative positional encoding makes attention sensitive to both direction and distance, which helps distinguish left and right context and neighboring entity spans.The relative encoding uses the target-context offset and enables attention scores to distinguish different directions and distances.
  • 3.2.2 Un-scaled Attention: The model removes the attention scaling factor because unscaled attention produces sharper distributions, which may suit sentences containing relatively few named entities.The implementation also avoids directly multiplying two learnable parameters and discards an output projection that is directly multiplied by a later layer.
  • 3.2.3 Conditional Random Field: All models use a CRF to capture dependencies between tags and Viterbi decoding to select the highest-probability valid label sequence.The CRF objective maximizes the probability of the gold label sequence.

4 Experiment

Experiments evaluate TENER across six NER datasets, comparing adapted Transformer components with BiLSTM, CNN, and vanilla Transformer baselines. The adapted model improves performance across Chinese and English settings, supports competitive character encoding, and converges as quickly as BiLSTM.

  • Results on Chinese NER Datasets: The vanilla Transformer underperforms BiLSTM- and CNN-based models on Chinese NER, while relative positional encoding substantially improves results.The adapted model outperforms BiLSTM across Chinese datasets, including the small Weibo dataset.
  • Results on Chinese NER Datasets: Removing attention scaling further improves Chinese NER, whereas scaled attention deteriorates performance.The reported comparison includes TENER with scaled attention as an ablation.
  • Results on English NER Datasets: On English NER, directional relative positional encoding and unscaled attention lift TENER above BiLSTM models and produce state-of-the-art results on two datasets.The comparison uses non-contextualized embeddings and excludes methods using pre-trained language models, pre-trained features, or higher-dimensional word vectors.
  • Analysis of Different Character Encoders: AdaTrans performs competitively with BiLSTM as a character encoder on CoNLL2003 and outpaces BiLSTM and CNN character encoders on OntoNotes 5.0.AdaTrans also gives the best word-level encoder performance across character-encoder combinations.
  • Convergent Speed Comparison: TENER converges as fast as BiLSTM and outperforms the vanilla Transformer on the OntoNotes 5.0 development set.The comparison includes BiLSTM, ID-CNN, Transformer, and TENER.

5 Conclusion

The conclusion presents TENER as a Transformer encoder customized for NER with direction-aware, distance-aware, and unscaled attention. Across six datasets, the modifications improve performance under matched embedding and external-knowledge conditions, and the adapted encoder also performs well for English character encoding.

  • 5 Conclusion: TENER customizes the Transformer encoder for NER with direction-aware, distance-aware, and unscaled attention.The model is designed to capture long-range context while adapting attention to NER.
  • 5 Conclusion: Across two English and four Chinese NER datasets, TENER outperforms previous models under the same pre-trained embeddings and external knowledge.The conclusion reports a performance increase across all six datasets.
  • 5 Conclusion: The adapted Transformer character encoder performs better than BiLSTM and CNN character encoders on two English NER datasets.The conclusion attributes this capability to extracting intricate character patterns.

6 Supplemental Material

The supplemental material specifies encoder hyperparameters and search ranges used for the experiments.

  • Hyperparameters: Character encoders use 30-dimensional randomly initialized character embeddings, with encoder-specific hidden sizes, kernels, heads, dropout, and feedforward dimensions.The adapted Transformer uses three heads of 10 dimensions each, 0.15 dropout, and a 60-dimensional feedforward layer.
  • Hyperparameter Search: Hyperparameters and search ranges are provided for ID-CNN, Transformer, and adapted Transformer encoders across Chinese and English NER datasets.The detailed settings are listed in Tables 6, 7, and 8.
Loading 1911.04474v3…