Source-linked AI summary
SpellGCN: Incorporating Phonological and Visual Similarities into Language Models for Chinese Spelling Check
Xingyi Cheng, Weidi Xu, Kunlong Chen, Shaohua Jiang, Feng Wang, Taifeng Wang, Wei Chu, Yuan Qi
TL;DR
Chinese Spelling Check requires correcting character errors while respecting semantic context and phonological or visual similarity, but prior methods often treated similarity as external candidate knowledge. SpellGCN builds similarity graphs whose learned character classifiers combine with BERT representations, achieving superior results on three benchmarks and large-margin gains over competitors.
Problem
Existing Chinese Spelling Check methods treat character-similarity knowledge as an external resource or heuristic rather than integrating it into language models.
Method
SpellGCN builds phonological and visual similarity graphs and combines their learned character classifiers with representations extracted by BERT in an end-to-end model.
Results
SpellGCN achieves the best results on three benchmark datasets and outperforms competitor models by a large margin.
Takeaways & Limitations
The experiments support incorporating phonological and visual similarity knowledge into language models for Chinese Spelling Check.
Takeaways & Limitations
Adapting SpellGCN to grammar error correction with insertion and deletion is left for future work.
Abstract
from arXiv · showhide
Chinese Spelling Check (CSC) is a task to detect and correct spelling errors in Chinese natural language. Existing methods have made attempts to incorporate the similarity knowledge between Chinese characters. However, they take the similarity knowledge as either an external input resource or just heuristic rules. This paper proposes to incorporate phonological and visual similarity knowledge into language models for CSC via a specialized graph convolutional network (SpellGCN). The model builds a graph over the characters, and SpellGCN is learned to map this graph into a set of inter-dependent character classifiers. These classifiers are applied to the representations extracted by another network, such as BERT, enabling the whole network to be end-to-end trainable. Experiments (The dataset and all code for this paper are available at https://github.com/ACL2020SpellGCN/SpellGCN) are conducted on three human-annotated datasets. Our method achieves superior performance against previous models by a large margin.
1 Introduction
Chinese Spelling Check must combine semantic context with phonological and visual character similarities, which commonly underlie Chinese spelling errors. SpellGCN integrates these similarities into an end-to-end language-model framework and outperforms prior systems on three benchmarks.
- Motivation: About 83% of Chinese spelling errors relate to phonological similarity and 48% to visual similarity.Chinese characters can be phonologically and visually similar while differing semantically, making contextual correction necessary.
- Approach: Two similarity graphs produce character representations that form classifiers applied to semantic representations extracted by BERT.The graph representations model interactions among similar characters while BERT supplies semantic capacity.
- Results: SpellGCN improves BERT and outperforms all competitor models by a large margin on three open benchmarks.The experiments evaluate the proposed model across three benchmark datasets.
- Approach: SpellGCN integrates pronunciation and shape similarities into semantic representations through specialized graph convolution and attentive combination operations.The method is designed as a novel end-to-end trainable model.
- Evaluation: The paper investigates the proposed graph convolution and attentive combination operations through quantitative and qualitative experiments.The contribution summary frames these components as objects of detailed investigation.
2 Related Work
Earlier CSC systems used language modeling, sequence tagging, or sequence-to-sequence generation, often treating character similarity as candidate-selection knowledge. SpellGCN instead models dependencies among similar characters with graph convolution alongside BERT.
- Prior CSC methods: Earlier CSC systems used unsupervised language models, discriminative sequence tagging, sequence-to-sequence models, and BERT.These approaches differ in how they detect and generate corrections, but earlier language models could not condition correction on the input sentence.
- Similarity knowledge: Confusion sets supplied similar-character candidates, which models then filtered or copied during correction.Similarity information was commonly used for candidate selection rather than modeled as relationships among characters.
- SpellGCN: SpellGCN uses graph convolution to model pronunciation and shape similarity and prior dependencies between characters, combined with BERT pretraining.The paper presents this as the first successful application of GCN to CSC.
3 Approach
SpellGCN combines pronunciation and shape similarity graphs with character representations from an extractor, using the resulting graph representations as target-character classifiers for CSC.
- Framework: The framework contains a character representation extractor and SpellGCN, which models inter-dependence among similar characters.The extractor produces character vectors, while SpellGCN outputs target vectors after similarity-based interactions.
- Similarity graphs: SpellGCN constructs separate pronunciation and shape graphs from a confusion set covering similar Chinese characters.The graphs are binary adjacency matrices whose edges indicate character pairs in the confusion set.
- Graph convolution: Graph convolution aggregates information from neighboring characters using normalized adjacency matrices and trainable transformations.BERT character embeddings initialize the node features, and the model omits nonlinear activation to preserve BERT’s semantic space.
- Graph combination: An attentive combination operation assigns character-specific weights to the pronunciation and shape graph representations.The attention weights use a shared learnable vector and a smoothness hyper-parameter β.
- CSC application: The model applies its classifiers to extractor representations and maximizes the log likelihood of target characters for CSC.BERT is used as the extractor in the described implementation, and the final character prediction supports correction and detection.
- Character classification: SpellGCN’s final representations serve as classifiers for characters in the confusion set, while extractor embeddings classify excluded vocabulary items.This design avoids modeling a graph over the entire extractor vocabulary for computational efficiency.
4 Experiments
Experiments evaluate SpellGCN on three SIGHAN benchmarks against established CSC baselines, using detection and correction metrics at character and sentence levels. SpellGCN consistently improves performance, while ablations show benefits from multiple graph layers and adaptive graph combination, and case studies illustrate pronunciation- and shape-constrained corrections.
- Datasets and metrics: Experiments use SIGHAN 2013, SIGHAN 2014, and SIGHAN 2015 test datasets, with training data totaling 10K manually collected samples plus 271K automatically generated samples.The evaluation reports precision, recall, and F1 for detection and correction at character and sentence levels.
- Baselines: SpellGCN is compared with five typical CSC systems, including LMC, SL, PN, FASpell, and BERT.BERT is trained under the comparable setting without SpellGCN.
- Main results: 9.2%, 9.7%, and 13.3% are the sentence-level correction F1 improvements over FASpell on SIGHAN 2013, 2014, and 2015, respectively.The comparison uses the C-F score, while FASpell was trained on different training data.
- Main results: SpellGCN consistently improves performance over BERT under both the PGNet/FASpell scripts and the official evaluation toolkit.On SIGHAN 2014 and 2015, FPR is 14.1% versus 15.3% and 13.2% versus 13.6% for SpellGCN and BERT, respectively.
- Ablation studies: Attention-based graph combination surpasses mean pooling, whereas sum pooling fails in the CSC task; the selected attention setting uses β = 3.The authors attribute the advantage to adaptive combination for each character node.
- Case study: Case studies show corrections constrained by pronunciation and shape, including replacing “麻坊” with “麻烦” and “音” with “影” in context.The examples illustrate how similarity information helps retrieve candidates that are semantically and phonologically or visually appropriate.
- Character embedding visualization: BERT embeddings capture semantic similarity but fail to model pronunciation similarity, whereas SpellGCN produces clusters reflecting pronunciation and shape relationships.The visualization uses t-SNE to show shape-similar character groups associated with “长” and “祀”.
5 Conclusions
SpellGCN incorporates phonological and visual similarities into language models for Chinese spelling checking, with effectiveness supported by empirical and analytical experiments. The approach may generalize to other prior-knowledge settings and languages, while grammar-error-correction adaptation remains future work.
- SpellGCN incorporates both phonological and visual similarities into language models for Chinese spelling checking.
- Empirical comparisons and analytical experiments verify SpellGCN’s effectiveness.
- SpellGCN can generalize to settings with specific prior knowledge and to other languages through analogous similarity graphs.
- Adapting SpellGCN to grammar error correction with insertion and deletion is left for future work.