Source-linked AI summary
Spelling Error Correction with Soft-Masked BERT
Shaohua Zhang, Haoran Huang, Jicong Liu, Hang Li
TL;DR
Chinese spelling error correction is challenging because accurate correction can require human-level language understanding, including world knowledge and inference, while BERT-based methods have limited error-detection capacity. The paper proposes Soft-Masked BERT, which connects an error-detection network to a BERT correction network through soft masking. On two datasets, it significantly outperforms baselines, and the authors identify inference-heavy errors as a remaining limitation.
Problem
Chinese spelling error correction is challenging because accurate correction can require human-level language understanding, including world knowledge and inference, while BERT-based methods have limited error-detection capacity.
Method
Soft-Masked BERT connects a network predicting character error probabilities to a BERT-based correction network through soft-masked character embeddings.
Results
Soft-Masked BERT significantly outperforms the state-of-the-art method solely utilizing BERT and other baselines on two datasets.
Takeaways & Limitations
The soft-masking technique is general and potentially useful for other detection-correction tasks.
Takeaways & Limitations
Among sampled test errors, 67% require strong reasoning or inference ability, and accurately correcting such typos requires stronger inference ability.
Abstract
from arXiv · showhide
Spelling error correction is an important yet challenging task because a satisfactory solution of it essentially needs human-level language understanding ability. Without loss of generality we consider Chinese spelling error correction (CSC) in this paper. A state-of-the-art method for the task selects a character from a list of candidates for correction (including non-correction) at each position of the sentence on the basis of BERT, the language representation model. The accuracy of the method can be sub-optimal, however, because BERT does not have sufficient capability to detect whether there is an error at each position, apparently due to the way of pre-training it using mask language modeling. In this work, we propose a novel neural architecture to address the aforementioned issue, which consists of a network for error detection and a network for error correction based on BERT, with the former being connected to the latter with what we call soft-masking technique. Our method of using `Soft-Masked BERT' is general, and it may be employed in other language detection-correction problems. Experimental results on two datasets demonstrate that the performance of our proposed method is significantly better than the baselines including the one solely based on BERT.
1 Introduction
Chinese spelling error correction is important for applications such as search, OCR, and essay scoring, yet difficult because it can require world knowledge and inference. Soft-Masked BERT addresses limitations in BERT-based correction by connecting error detection with BERT correction through soft masking.
- Chinese spelling error correction supports applications including search, optical character recognition, and essay scoring.
- World knowledge and inference are required to resolve context-dependent Chinese spelling errors that may form valid but unintended words.Examples distinguish 金字塔 from 金子塔 and 求生欲 from the valid alternative 求胜欲.
- BERT-based correction predicts the most likely character from candidate lists after pre-training and supervised fine-tuning, but tends to make no correction because its pre-training provides limited error-detection capacity.The method uses character-level BERT, data augmentation, and candidate prediction at each sentence position.
- Soft-Masked BERT combines a Bi-GRU error-detection network with a BERT-based correction network, using predicted error probabilities to soft-mask character embeddings.The correction network is jointly trained with the detector, and soft masking becomes hard masking when the error probability equals one.
- Experiments on SIGHAN and the ten-times-larger News Title dataset show that Soft-Masked BERT significantly outperforms baselines in accuracy measures.
- The work contributes a novel CSC architecture and empirical verification of its effectiveness.
2 Our Approach
Soft-Masked BERT formulates CSC as sequence labeling and combines a Bi-GRU detection network with a BERT-based correction network through soft masking. The model is trained end-to-end with separate detection and correction objectives.
- 2.1 Problem and Motivation: CSC transforms an input character sequence into an equal-length corrected sequence, usually copying most characters and replacing only errors.
- 2.1 Problem and Motivation: BERT-based CSC may overproduce non-corrections because pre-training masks only 15% of characters, limiting error-detection capacity.
- 2.2 Model: Soft-Masked BERT uses a Bi-GRU detection network to estimate each character’s error probability and a BERT-based correction network to predict replacements.
- 2.2 Model: Soft masking forms each position’s embedding as a probability-weighted combination of its input embedding and the mask embedding.Higher error probability makes the resulting embedding closer to the mask embedding; lower probability keeps it closer to the input embedding.
- 2.4 Correction Network: The correction network processes soft-masked embeddings with BERT, uses a residual connection, and selects the highest-probability candidate at each position.
- 2.5 Learning: Soft-Masked BERT is trained end-to-end by jointly optimizing objectives for error detection and error correction.Training data consists of original and corrected sequence pairs, which can be generated with a confusion table.
3 Experimental Results
Experiments on SIGHAN and News Title show that Soft-Masked BERT outperforms the baselines, while analyses examine training size, masking design, hyperparameters, and remaining error types.
- Datasets: 15,730 texts comprise the News Title dataset, including 5,423 texts containing errors across 3,441 error types.The dataset is divided into test and development sets of 7,865 texts each; SIGHAN contains 1,100 texts and 461 error types.
- Main results: Soft-Masked BERT significantly outperforms baseline methods on both SIGHAN and News Title in accuracy measures.On News Title, it performs better than the baselines on all reported measures.
- Main results: 54% recall and 55% precision are exceeded for correction level on News Title.These are reported as the best correction-level results on that dataset.
- Main results: Soft-Masked BERT beats BERT-Finetune by large margins on both datasets, whereas BERT-Pretrain performs fairly poorly.The results indicate that fine-tuning improves BERT performance and that error detection contributes to the improvement.
- Training-data analysis: Soft-Masked BERT performs best with 5 million training examples and remains consistently superior to BERT-Finetune across training-data sizes.The authors report higher performance with more training data in the tested range.
- Ablation study: All major components of Soft-Masked BERT are necessary for high performance, and λ = 0.8 yields the highest F1 score.The ablation compares residual removal, hard masking, and randomized masking; λ controls the relative weight on error correction.
- Error analysis: 67% of sampled errors require strong reasoning ability, while 11% are attributed to insufficient world knowledge.The remaining 22% have no significant type; examples involve ambiguous corrections and unfamiliar proper names.
4 Related Work
CSC research spans traditional and deep-learning approaches, including pipelines, sequential labeling, language models, rules, and Seq2Seq systems. BERT-based methods extend language representation modeling to spelling correction.
- Early CSC systems used pipelines for error detection, candidate generation, and final candidate selection.
- Unsupervised language models, rules, conditional random fields, and hidden Markov models were applied to CSC.
- Deep-learning approaches include Seq2Seq models that transform input sentences into corrected sentences, including one using BERT as encoder.
- BERT uses a Transformer encoder, large-corpus self-supervised pre-training, and task-specific fine-tuning, and was extended here to Soft-Masked BERT.
5 Conclusion
The paper proposes Soft-Masked BERT for Chinese spelling error correction, combining error detection with BERT-based correction. Across two datasets, it significantly outperforms the state-of-the-art BERT-only method, while soft-masking may generalize to other detection-correction tasks.
- Soft-Masked BERT combines a detection network that identifies likely incorrect characters with a BERT-based correction network.
- The detection network soft-masks likely errors before the correction network predicts corrected characters.
- Soft-Masked BERT significantly outperforms the state-of-the-art method that solely utilizes BERT on two datasets.
- Soft-masking is presented as a general technique potentially useful in other detection-correction tasks.