Source-linked AI summary
Glyce: Glyph-vectors for Chinese Character Representations
Yuxian Meng, Wei Wu, Fei Wang, Xiaoya Li, Ping Nie, Fan Yin, Muyu Li, Guoyin Wang, Qinghong Han, Xiaofei Sun, Jiwei Li
TL;DR
Chinese glyph information is difficult to exploit because modern scripts lose pictographic evidence and standard vision models generalize poorly on character images. Glyce combines historical and contemporary scripts, a Tianzige-CNN, and auxiliary image classification to model Chinese characters as images. Glyph-based models consistently outperform word/character ID-based models across many Chinese NLP tasks and achieve state-of-the-art results.
Problem
Chinese glyph information is difficult to exploit because scripts lose pictographic evidence and standard computer-vision models generalize weakly on character data.
Method
Glyce models Chinese characters as images using historical and contemporary scripts, Tianzige-CNN structures, and auxiliary image classification in multi-task learning.
Results
Glyce achieves state-of-the-art performance across a wide range of Chinese NLP tasks, including tagging, classification, parsing, and semantic role labeling.
Takeaways & Limitations
Glyph-based representations provide a general way to model character semantics and can be integrated into existing deep-learning systems.
Abstract
from arXiv · showhide
It is intuitive that NLP tasks for logographic languages like Chinese should benefit from the use of the glyph information in those languages. However, due to the lack of rich pictographic evidence in glyphs and the weak generalization ability of standard computer vision models on character data, an effective way to utilize the glyph information remains to be found. In this paper, we address this gap by presenting Glyce, the glyph-vectors for Chinese character representations. We make three major innovations: (1) We use historical Chinese scripts (e.g., bronzeware script, seal script, traditional Chinese, etc) to enrich the pictographic evidence in characters; (2) We design CNN structures (called tianzege-CNN) tailored to Chinese character image processing; and (3) We use image-classification as an auxiliary task in a multi-task learning setup to increase the model's ability to generalize. We show that glyph-based models are able to consistently outperform word/char ID-based models in a wide range of Chinese NLP tasks. We are able to set new state-of-the-art results for a variety of Chinese NLP tasks, including tagging (NER, CWS, POS), sentence pair classification, single sentence classification tasks, dependency parsing, and semantic role labeling. For example, the proposed model achieves an F1 score of 80.6 on the OntoNotes dataset of NER, +1.5 over BERT; it achieves an almost perfect accuracy of 99.8\% on the Fudan corpus for text classification. Code found at https://github.com/ShannonAI/glyce.
1 Introduction
Glyce addresses inconsistent use of Chinese glyph information by combining richer historical scripts, a character-tailored CNN, and auxiliary image classification. The resulting glyph-based models achieve strong performance across Chinese NLP tasks.
- Motivation: Chinese characters encode semantic information through their logographic glyphs, motivating glyph-aware representations for Chinese NLP.
- Problem: Earlier CNN-based character representations showed inconsistent gains because they used limited scripts, unsuitable architectures, and lacked regulatory training objectives.Historical script evolution reduced pictographic information, while character images are much smaller than typical ImageNet inputs.
- Approach: Glyce treats Chinese characters as images and uses CNNs to obtain their representations.
- Innovations: Glyce combines historical and contemporary scripts, varied writing styles, Tianzige-CNN structures, and image-classification loss in multi-task learning.These techniques target pictographic enrichment, character-image modeling, and generalization.
- Results: Glyce improves a wide range of Chinese NLP tasks and obtains state-of-the-art results in tagging, classification, parsing, and semantic role labeling.
2 Glyce
Glyce builds Chinese character representations from historical and contemporary glyph images, using a Tianzige-CNN and auxiliary image classification. It can also combine glyph embeddings with pretrained BERT representations for downstream NLP tasks.
- 2 Glyce: Historical scripts and varied writing styles enrich pictographic evidence from Chinese character images.The approach includes bronzeware, clerical, seal, traditional, and cursive scripts.
- 2 Glyce: Tianzige-CNN is tailored to Chinese character images, whose small size and limited character inventory make standard deep CNNs perform poorly.The architecture uses convolution and pooling to form a 2 × 2 structure representing radical arrangement.
- 2 Glyce: Glyce adds image classification as an auxiliary objective, training CNN-derived glyph embeddings to predict each character's charID.This auxiliary task is intended to improve generalization and prevent overfitting.
- 2 Glyce: The final objective linearly combines the task-specific loss with the auxiliary image-classification loss.The weight λ(t) controls their trade-off and decreases the auxiliary objective's influence during training.
- 2 Glyce: Glyph embeddings can feed downstream RNNs, LSTMs, and transformers, or be integrated with BERT through a four-layer Glyce-BERT architecture.The architecture includes BERT, glyph, Glyce-BERT, and task-specific output layers.
3 Tasks
Glyce-BERT applies glyph-enhanced representations across sequence labeling, single-sentence classification, and sentence-pair classification tasks. Task-specific output layers adapt the shared representations to each prediction setting.
- 3 Tasks: Glyce-BERT supports multiple NLP task families, including sequence labeling and sentence-level classification.The model is described for different tasks in the task-specific integration framework.
- 3 Tasks: Sequence labeling tasks such as NER, CWS, and POS use a CRF to predict one label for each character.The CRF receives embeddings from the Glyce-BERT task-specific layer.
- 3 Tasks: Single-sentence classification feeds the task-specific CLS representation to a softmax layer for sentence-level label prediction.This follows the analogous BERT strategy using the final CLS representation.
- 3 Tasks: Sentence-pair classification concatenates [CLS, s1, SEP, s2, SEP] and uses the resulting CLS representation for label prediction.Glyce-BERT applies the same interaction format while incorporating glyph-enhanced representations.
4 Experiments
Glyce is evaluated across tagging, sentence classification, dependency parsing, and semantic role labeling, consistently improving strong baselines and setting new state-of-the-art results.
- Tagging: Glyce-BERT outperforms BERT across all NER, CWS, and POS datasets, while Lattice-Glyce outperforms all non-BERT models.The paper attributes BERT’s exception on Weibo to mismatch between its Wikipedia pretraining data and Weibo.
- Sentence Pair Classification: BERT+Glyce sets new state-of-the-art results on all four sentence-pair classification benchmarks.Glyce+BiMPM also outperforms the non-BERT BiMPM baseline, while BERT outperforms all non-BERT models.
- Single Sentence Classification: 99.8 accuracy is achieved by Glyce+BERT on the Fudan corpus, alongside state-of-the-art results on all evaluated single-sentence classification datasets.The datasets are ChnSentiCorp, the Fudan corpus, and Ifeng.
- Dependency Parsing and Semantic Role Labeling: +0.9 UAS and +0.8 LAS are obtained over previous state-of-the-art dependency parsing results with Glyce-word embeddings.For semantic role labeling, Glyce reaches an F1 score of 83.7, improving previous state-of-the-art performance by 0.9.
5 Ablation Studies
Ablations show that training strategy, output-layer architecture, auxiliary image classification, and Tianzige-CNN design materially affect Glyce’s performance.
- Training Strategy: BERT-glyce-joint outperforms the Glyph-Joint and direct joint-training strategies.The authors explain that direct joint training can let the pretrained BERT layer be misled early by the randomly initialized glyph layer.
- Structures of the task-specific output layer: Transformer output layers outperform BiLSTMs, CNNs, and BiMPM in the tested sentence-pair classification setting.The authors conjecture that this reflects consistency between the transformer output layer and BERT.
- Auxiliary Image Classification: +0.8 performance boost is obtained by adding the auxiliary image-classification training objective.The objective trains glyph representations through an image-classification task in addition to the task-specific objective.
- CNN structures: +1.0 F1 is obtained with the Tianzige-CNN structure compared with alternative CNN structures.The paper links poor direct deep-CNN performance to small character images and the limited number of distinct Chinese characters.
6 Conclusion
Glyce models Chinese characters as images, uses Tianzige-CNN to extract character semantics, and can be integrated into existing deep learning systems.
- Conclusion: Glyce treats Chinese characters as images and uses Tianzige-CNN to extract character semantics.The paper presents Glyce as a general way to model character semantics in logographic languages.
- Conclusion: Glyph embeddings can be integrated into existing deep learning systems like word embeddings.The conclusion describes Glyce as general and fundamental and states that it can be integrated into any existing deep learning system.