PaperlayerView exact revision on arXiv

Source-linked AI summary

CAN-NER: Convolutional Attention Network for Chinese Named Entity Recognition

Yuying Zhu, Guoxin Wang, Börje F. Karlsson

arXiv:1904.02141v3cs.CL

TL;DR

Chinese NER faces ambiguous word boundaries that complicate word-based recognition and expose models to OOV and lexicon dependencies. CAN combines character-level local convolutional attention with Bi-GRU global self-attention, and reports state-of-the-art results across datasets without external word embeddings or lexicons.

  • Problem

    Chinese word-boundary ambiguity complicates word-based NER, and character embeddings alone lose word and word-sequence information.

    Source: §I · p. 1

  • Method

    CAN combines character-based convolutional local attention with a Bi-GRU and global self-attention for local character features and sentence context.

    Source: §I · p. 2, B. Convolutional Attention Layer · p. 2

  • Results

    CAN reports state-of-the-art F1 without external word embeddings or lexicons across Weibo, Chinese Resume, and MSRA datasets.

    Source: §I · p. 2, 1) Weibo Dataset · p. 4, 2) Chinese Resume Dataset · p. 4

  • Takeaways & Limitations

    The authors describe the resource-light model as more practical for real NER systems.

    Source: §I · p. 2

Abstract

from arXiv · show

Named entity recognition (NER) in Chinese is essential but difficult because of the lack of natural delimiters. Therefore, Chinese Word Segmentation (CWS) is usually considered as the first step for Chinese NER. However, models based on word-level embeddings and lexicon features often suffer from segmentation errors and out-of-vocabulary (OOV) words. In this paper, we investigate a Convolutional Attention Network called CAN for Chinese NER, which consists of a character-based convolutional neural network (CNN) with local-attention layer and a gated recurrent unit (GRU) with global self-attention layer to capture the information from adjacent characters and sentence contexts. Also, compared to other models, not depending on any external resources like lexicons and employing small size of char embeddings make our model more practical. Extensive experimental results show that our approach outperforms state-of-the-art methods without word embedding and external lexicon resources on different domain datasets including Weibo, MSRA and Chinese Resume NER dataset.

I. INTRODUCTION

Chinese NER is complicated by ambiguous segmentation, OOV words, and dependence on word embeddings and lexicons. CAN combines character-based local convolutional attention with a Bi-GRU and global self-attention, reporting state-of-the-art F1 without external embeddings or lexicons.

I. INTRODUCTION

  • Segmentation ambiguity can change sentence meaning and entity boundaries, while word-based models face OOV and lexicon-cost challenges.Large lexicons require memory and matching time, and word embeddings increase model size.Source: §I · p. 1
  • Concatenating segmentation vectors with character embeddings lets convolutional attention group meaning-related characters and reduce segmentation-error impact.The paper contrasts this with earlier approaches that add segmentation features to the last character embedding or jointly learn segmentation and NER.Source: §I · p. 2
  • The convolutional attention layer improves F1 over a baseline with a normal CNN layer.The introduction reports the comparison without giving a numeric difference.Source: §I · p. 2

II. METHODOLOGY

  • CAN combines character-based CNN local attention with Bi-GRU global self-attention to capture local character relations and sentence context.The architecture is described as modeling both character- and word-level information.Source: §I · p. 2, §II · p. 2

I. INTRODUCTION

  • The proposed character-based model achieves state-of-the-art F1 without external word embeddings or lexicon resources.The paper presents this resource-light design as more practical for real NER systems.Source: §I · p. 2

A. Formulation

The formulation maps character-embedding sequences to per-character entity labels. The convolutional attention layer combines character, segmentation, and position information within local windows, then applies convolution and sum pooling.

A. Formulation

  • The task learns a function from an input sentence's character embeddings to per-character entity labels, including O.The sentence and label sequences are denoted X_i and Y_i, respectively.Source: A. Formulation · p. 2

B. Convolutional Attention Layer

  • Each character representation concatenates its character embedding with a BMES segmentation mask.The mask encodes segmentation information.Source: B. Convolutional Attention Layer · p. 2
  • The full architecture uses convolutional attention for character- and word-level information, then BiGRU-CRF with global self-attention for long-range sentence relations.The figure caption describes these as the roles of the two stages.Source: Figure 2 · p. 2
  • For each local window, the model adds position embeddings and applies attention to center-context relations before convolution and sum pooling.Position embeddings identify each character's position within the window.Source: B. Convolutional Attention Layer · p. 2
  • Local attention produces hidden vectors for the positions in the window, supporting implicit grouping of meaning-related characters.The convolutional attention layer is intended to encode the character sequence and local context.Source: B. Convolutional Attention Layer · p. 2
  • The local attention weights are denoted α_m and are computed using a score function s.The score function is defined over character representations.Source: B. Convolutional Attention Layer · p. 3
  • The CNN uses d_h kernels over a k-token context window and sum-pools across the first dimension.The convolution parameters include W_c and b_c.Source: B. Convolutional Attention Layer · p. 3

C. BiGRU-CRF with Global Attention

The model feeds convolutional-attention features into a BiGRU-CRF, combining local character context with sequential and sentence-level information to predict character labels.

C. BiGRU-CRF with Global Attention

  • Convolutional-attention features feed a BiGRU-CRF model to predict the final label for each character.The architecture combines character-, word-, and sentence-level information through multi-feature embeddings, local attention, and global self-attention.Source: C. BiGRU-CRF with Global Attention · p. 3
  • The BiGRU models sequential sentence information, while global self-attention is applied to handle sentence-level information.The global attention weights are defined over character positions in the sentence.Source: C. BiGRU-CRF with Global Attention · p. 3
  • A standard CRF is placed on the concatenated BiGRU and global-attention outputs, and Viterbi decoding produces the predicted tag sequence.The CRF computes the probability of the ground-truth label sequence.Source: C. BiGRU-CRF with Global Attention · p. 3

D. Training

Training uses a log-likelihood objective, with shuffled training instances processed in batches and AdaDelta used for optimization.

D. Training

  • The model is trained with a log-likelihood objective over the training examples.The passage identifies this objective as the loss function.Source: D. Training · p. 3
  • Training instances are shuffled at each iteration and fed to the model in batch updates.Source: D. Training · p. 3
  • AdaDelta [13] is used to optimize the final objective.Source: D. Training · p. 3

III. EXPERIMENTS

The experiments evaluate Chinese NER on datasets from different domains using standard precision, recall, and F1 metrics.

III. EXPERIMENTS

  • The experiments use Chinese NER datasets from different domains to demonstrate the proposed model's effectiveness.Source: §III · p. 3
  • Standard precision (P), recall (R), and F1-score (F1) are the evaluation metrics.Source: §III · p. 3

A. Experimental Settings

The experimental settings cover news, social-media, and resume datasets, with dataset-specific entity annotations and preprocessing; model hyperparameters are selected using development-set performance.

A. Experimental Settings

  • The evaluation uses OntoNotes 4 and MSRA for news, Weibo for social media, and Chinese Resume for additional test-domain variety.Source: A. Experimental Settings · p. 3
  • Weibo and OntoNotes use four entity categories, Chinese Resume uses eight, and MSRA uses three; MSRA has no development subset.Weibo includes named and nominal mentions.Source: A. Experimental Settings · pp. 3–4
  • Table II presents Weibo NER results.Source: Table II · p. 4
  • For Weibo, Chinese Resume, and MSRA test sections without gold segmentation, the authors automatically segment the data and use BIOES tagging.Source: A. Experimental Settings · p. 4
  • Character embeddings and CNN and Bi-GRU hidden sizes are 300 dimensions, the CNN window is 5, and the initial AdaDelta learning rate is 0.005.Hyperparameters are adjusted according to Chinese NER development-set performance; character embeddings come from.Source: A. Experimental Settings · p. 4

B. Experimental Results

The experiments compare CAN with previous state-of-the-art methods on four Chinese NER datasets and evaluate two baselines alongside CAN.

B. Experimental Results

1) Weibo Dataset:

CAN achieves the highest reported Weibo F1-score in the comparison and a state-of-the-art result on Chinese Resume. The section also places these results alongside methods using external data or lexicons.

1) Weibo Dataset:

  • Reported overall F1-scores span 54.82%–58.99%; model, using cross-domain and semi-supervised data, improves from 54.82% to 58.23% versus, while CWS-joint model [18] reaches 58.99%.The jointly trained NER model reports 56.05%; the adversarial CWS transfer model reports 58.70%, and the lexicon-integrating model [10] reports 58.79%.Source: 1) Weibo Dataset · p. 4
  • 59.31% F1-score is the highest result among existing models on Weibo.Replacing the normal CNN with convolutional attention raises the score from 55.91%.Source: 1) Weibo Dataset · p. 4

2) Chinese Resume Dataset:

3) OntoNotes Dataset:

On OntoNotes 4, CAN reaches a competitive character-based F1-score without external data; on MSRA, it is state of the art among character-based models. The comparisons include earlier methods using additional features or lexicons.

3) OntoNotes Dataset:

  • [25] reports OntoNotes F1-score improvement from 68.57% to 76.40% using neural and discrete features.The cited comparison also notes bilingual-data results of 74.32% for and 73.88% for.Source: 3) OntoNotes Dataset · p. 4
  • 73.64% F1-score is competitive among character-based models on OntoNotes 4 without external data.Source: 3) OntoNotes Dataset · p. 5, Table IV · p. 5

4) MSRA Dataset:

  • 92.97% F1-score is competitive with [10]'s 93.18% MSRA result, and CAN is state of the art among character-based models.The paper's character-based method uses no additional lexicon data or word embedding information.Source: 4) MSRA Dataset · p. 5, Table V · p. 5

C. Discussion

CAN’s attention mechanisms show local character-to-character phrase links and long-distance sentence-context links, while the model outperforms previous work on Weibo and Chinese Resume and is competitive on MSRA and OntoNotes 4 without external resources. The authors note weaker performance on OntoNotes 4 and describe neural-network approaches to context and representation learning.

1) Effectiveness of Convolutional Attention and Global Self-Attention:

  • Attention visualizations and experimental results support Convolutional Attention’s effectiveness in extracting phrase information between adjacent characters.The examples link characters in “美国,” “克林顿,” and “欧洲.”Source: 1) Effectiveness of Convolutional… · p. 5
  • Global self-attention captures long-distance sentence context, with “克林顿” attending more strongly to predicate “前往” than time phrase “1号.”The authors say this helps overcome the limitation of recurrent neural networks.Source: 1) Effectiveness of Convolutional… · p. 5

2) Results Analysis:

  • The model outperforms previous work on Weibo and Chinese Resume and gains competitive results on MSRA and OntoNotes 4 without external resources; OntoNotes 4 performance is relatively unsatisfying.The authors suggest discrete features and external resources may have more influence on OntoNotes 4 when training data is insufficient.Source: 2) Results Analysis · p. 5

A. Neural Network Models

  • Neural NER approaches use CNNs and LSTM-CRF architectures for context, character-level representations, and integration of word- and character-level information.The cited work also includes character-level pretraining and co-training, as well as deep language representations such as ELMo and BERT.Source: A. Neural Network Models · pp. 5–6

B. Attention Mechanism related Models

Prior attention-based NER work uses self-attention for long-range dependencies and combines character and word representations, while Chinese NER methods address boundary information through varied approaches. CAN combines local-attention CNN and global self-attention Bi-GRU layers, and the authors propose joint segmentation and NER learning as future work.

B. Attention Mechanism related Models

  • Prior attention approaches include self-attention for long-range dependencies, attention-based integration of character and word representations, and co-attention using tweet images.The passage notes that image-related resources are insufficient beyond websites with text-related images.Source: B. Attention Mechanism related Models · p. 6

C. Chinese NER

  • Chinese NER work has addressed boundary information through feature-based CRFs, character representations, joint segmentation, cross-domain learning, lexicons, and crowd annotations.One approach matches text words with a lexicon in a word-character lattice rather than segmenting first.Source: C. Chinese NER · p. 6

V. CONCLUSION

  • CAN combines local-attention CNN and global self-attention Bi-GRU layers to capture word-level features and context, and the conclusion reports outperforming state-of-the-art systems across domain datasets.The model is designed to avoid word-embedding and additional lexicon dependencies.Source: §V · p. 6
  • The authors identify joint learning of word segmentation and NER as future work to further reduce an unspecified quantity or issue.The supplied conclusion ends before specifying what is to be reduced.Source: §V · p. 6
Loading 1904.02141v3…