Source-linked AI summary
FLAT: Chinese NER Using Flat-Lattice Transformer
Xiaonan Li, Hang Yan, Xipeng Qiu, Xuanjing Huang
TL;DR
Chinese NER lattice models can use word information but often process complex dynamic structures inefficiently. FLAT converts the lattice into spans with position encoding and applies a Transformer, outperforming other lexicon-based models in performance and efficiency across four datasets. The paper leaves adaptation to different lattice or graph types for future work.
Problem
Existing lattice-based Chinese NER models have difficulty fully using GPU parallel computation and usually have low inference speed.
Method
FLAT converts a character-word lattice into spans representing characters or latent words with their original lattice positions, then models them with a Transformer and span position encoding.
Results
FLAT outperforms baseline and other lexicon-based models on four Chinese NER datasets, improving average F1 by 1.72 over TENER and 1.51 over lattice LSTM.
Takeaways & Limitations
FLAT combines lattice information with parallel Transformer computation and achieves stronger performance and inference efficiency than other lexicon-based models.
Takeaways & Limitations
The paper leaves adapting FLAT to different kinds of lattices or graphs for future work.
Abstract
from arXiv · showhide
Recently, the character-word lattice structure has been proved to be effective for Chinese named entity recognition (NER) by incorporating the word information. However, since the lattice structure is complex and dynamic, most existing lattice-based models are hard to fully utilize the parallel computation of GPUs and usually have a low inference-speed. In this paper, we propose FLAT: Flat-LAttice Transformer for Chinese NER, which converts the lattice structure into a flat structure consisting of spans. Each span corresponds to a character or latent word and its position in the original lattice. With the power of Transformer and well-designed position encoding, FLAT can fully leverage the lattice information and has an excellent parallelization ability. Experiments on four datasets show FLAT outperforms other lexicon-based models in performance and efficiency.
1 Introduction
Chinese NER must handle word-segmentation information, motivating lattice-based models that incorporate latent words. FLAT replaces complex lattice processing with a span-based Transformer representation designed for performance and inference efficiency.
- Chinese NER is more difficult than English NER because it usually involves word segmentation.
- Lattice structures incorporate word information and help avoid error propagation from word segmentation.
- Lattice-based models represent characters and latent words in a directed acyclic graph whose word boundaries determine position.
- Existing lattice-compatible RNN and CNN methods struggle with long-distance dependencies and cannot fully exploit GPU parallel computation.
- FLAT assigns head and tail positional indices to characters or words, allowing a Transformer to model lattice input directly.
- FLAT is reported to outperform other lexicon-based methods in both performance and inference speed.
2 Background
Transformer encoders use self-attention and feedforward layers to model sequence representations. FLAT's background design replaces absolute sequential position encoding with relative span-position information for lattice inputs.
- A Transformer encoder consists of self-attention and feedforward network layers, with residual connections and layer normalization after each sublayer.
- Multi-head self-attention computes separate attention results and concatenates them across H heads.
- The token representation E is projected into query, key, and value representations using learnable parameter matrices.
- The vanilla Transformer uses absolute position encoding, whereas FLAT considers relative lattice positions important for NER.
3 Model
FLAT converts a lattice into recoverable spans and uses relative head–tail distances to encode span relations for Transformer self-attention.
- 3.1 Converting Lattice into Flat Structure: Each flat-lattice span represents a character or word with head and tail indices inherited from the original sequence.Characters have identical head and tail indices; words span their first and last character positions.
- 3.1 Converting Lattice into Flat Structure: The transformation is recoverable: character spans reconstruct the sequence, while word spans form skip-paths that preserve the lattice structure.
- 3.2 Relative Position Encoding of Spans: FLAT represents relations between spans with four relative distances derived from their heads and tails.These distances capture intersection, inclusion, separation, and finer positional information between characters and words.
- 3.2 Relative Position Encoding of Spans: A learnable nonlinear transformation concatenates position encodings of the four distances to produce the final relative span-position representation.The position function follows Vaswani et al. (2017), and the resulting encoding is used in a variant of self-attention.
- 3 Model: After Transformer encoding, FLAT sends only character representations to an output layer followed by a CRF.
4 Experiments
Experiments evaluate FLAT across four Chinese NER datasets, efficiency settings, structural ablations, and compatibility with BERT. FLAT achieves stronger average F1 and inference efficiency than comparison models, while its BERT gains depend on dataset size.
- Experimental Setup: Four Chinese NER datasets—Ontonotes 4.0, MSRA, Resume, and Weibo—are evaluated using BiLSTM-CRF, TENER, and lexicon-based baselines.The experiments use the same train, development, and test split as Gui et al. (2019b), with one Transformer encoder layer for FLAT.
- Overall Performance: 1.72 average F1 points over TENER, 1.51 over lattice LSTM, and 0.73 over CGN are reported for FLAT.The paper states that FLAT outperforms baseline and other lexicon-based models on all four datasets; gains are less significant on smaller datasets.
- Advantage of Fully-Connected Structure: Masking character-to-self-matched-word attention significantly deteriorates FLAT, whereas masking attention beyond distance 10 degrades performance only slightly.The ablation supports the importance of self-matched word information and the model’s use of longer-distance dependencies.
- Efficiency of FLAT: 4.97× versus 2.1× is the batch-parallel speedup for FLAT versus lattice LSTM at batch size 16.FLAT has no recurrent module and fully leverages GPU parallel computation; GNN-based models remain slowed by their RNN encoders.
- How FLAT Brings Improvement: Span F measures span correctness alone, while Type Acc is the proportion of fully correct predictions among span-correct predictions.FLAT’s improvement over TENER is more pronounced for Span F than Type Acc; FLAThead deteriorates more on Span F than Type Acc.
- Compatibility with BERT: FLAT+BERT significantly improves over BERT on Ontonotes and MSRA but yields marginal improvement on Resume and Weibo.The comparison uses BERT+MLP+CRF as BERT and FLAT with fine-tuned BERT embeddings.
5 Related Work
Prior lattice-based approaches either adapt recurrent or convolutional models to lattice inputs or encode lattices with graph neural networks. FLAT instead represents lattice position with recoverable span-based head and tail information, differing from other lattice Transformers and porous representations.
- Lattice-based NER: Lattice LSTM and LR-CNN directly encode lattice inputs, while LGN and CGN convert lattices into graphs for graph neural networks.The former methods suffer from low inference efficiency or limited long-distance modeling, while graph-based approaches retain sequential components in some designs.
- Lattice-based Transformer: Lattice-based Transformers differ mainly in how they represent lattice structure, including absolute positions, node relations, or distances from lattice nodes.Prior approaches use first-character positions, longest distances to the start node, or shortest distances between nodes.
- Lattice-based Transformer: FLAT uses head and tail span positions that can map to the three prior position-description schemes, whereas the reverse mapping is not always possible.The paper argues that this span encoding is more natural for NER because NER is more sensitive to position information than translation.
- Lattice-based Transformer: Porous Lattice Transformer combines head positions, relative relations, an extra GRU, and porous attention, whereas FLAT uses head-tail spans without those components.The paper states that the porous model’s head-plus-relative-relation representation can lose position information, while FLAT’s position information is recoverable.
6 Conclusion and Future Work
FLAT converts lattice structure into spans with specialized position encoding to incorporate lexicon information for Chinese NER, achieving improved performance and efficiency over other lexicon-based models. Adapting the model to other lattice or graph types remains future work.
- FLAT converts lattice structure into a set of spans and introduces specific position encoding for Chinese NER.
- Experimental results show FLAT outperforms other lexicon-based models in performance and efficiency.
- The authors leave adapting FLAT to different kinds of lattice or graph as future work.
7 Appendices
The appendices describe dataset-specific hyper-parameter selection: development experiments determine settings for Ontonotes and MSRA, while random search is used for Resume and Weibo.
- Ontonotes development experiments determine the hyper-parameters used for Ontonotes and MSRA.
- Random search determines the optimal hyper-parameters for the smaller Resume and Weibo datasets.
- Table 5 lists hyper-parameters for Ontonotes and MSRA, while Table 6 lists random-search ranges for Weibo and Resume.