PaperlayerView exact revision on arXiv

Source-linked AI summary

Read, Listen, and See: Leveraging Multimodal Information Helps Chinese Spell Checking

Heng-Da Xu, Zhongli Li, Qingyu Zhou, Chao Li, Zizhen Wang, Yunbo Cao, Heyan Huang, Xian-Ling Mao

arXiv:2105.12306v1cs.CL

TL;DR

Chinese spelling errors can involve characters with similar meanings, pronunciations, or shapes, motivating the use of phonetic and graphic information alongside semantics. ReaLiSe encodes these modalities and selectively fuses them; on SIGHAN benchmarks, it outperforms prior state-of-the-art models, including SpellGCN.

  • Problem

    Chinese spelling errors include semantically, phonetically, or graphically similar characters, so CSC needs phonetic and graphic knowledge alongside textual semantics.

    Source: §1 · pp. 1–2

  • Method

    ReaLiSe uses tailored semantic, phonetic, and graphic encoders with selective modality fusion, and adds acoustic and visual pretraining tasks.

    Source: §1 · p. 2, §3.4 · p. 5, §5 · p. 9

  • Results

    ReaLiSe improves over SpellGCN by an average of 2.4% F1 at detection level and 2.6% F1 at correction level on SIGHAN benchmarks.

    Source: §1 · p. 2

  • Takeaways & Limitations

    SIGHAN results support the conclusion that leveraging acoustic and visual information helps Chinese spell checking.

    Source: §5 · p. 9

    SIGHAN13 annotation omits many mixed uses of 的, 地, and 得, so a well-performing model may receive a poor score on that test set.

    Source: §4.2 · p. 6

Abstract

from arXiv · show

Chinese Spell Checking (CSC) aims to detect and correct erroneous characters for user-generated text in the Chinese language. Most of the Chinese spelling errors are misused semantically, phonetically or graphically similar characters. Previous attempts noticed this phenomenon and try to use the similarity for this task. However, these methods use either heuristics or handcrafted confusion sets to predict the correct character. In this paper, we propose a Chinese spell checker called ReaLiSe, by directly leveraging the multimodal information of the Chinese characters. The ReaLiSe model tackles the CSC task by (1) capturing the semantic, phonetic and graphic information of the input characters, and (2) selectively mixing the information in these modalities to predict the correct output. Experiments on the SIGHAN benchmarks show that the proposed model outperforms strong baselines by a large margin.

1 Introduction

Chinese spelling errors can involve characters that are semantically, phonetically, or graphically similar, so context alone may not capture the relevant cues. REALISE combines textual, phonetic, and graphic information through selective fusion, and reports better SIGHAN results than prior methods, including on errors absent from handcrafted confusion sets.

1 Introduction

  • Chinese spelling errors include semantically, phonetically, or graphically similar characters, rather than only words that are invalid in computer systems.The examples include “平” and “瓶,” which share a pronunciation, and “轻” and “经,” which have similar pronunciations and a shared radical.Source: §1 · pp. 1–2
  • The paper argues that CSC should use phonetic and graphic character knowledge alongside textual semantics.Source: §1 · p. 2
  • REALISE uses BERT for textual semantics, pinyin features for phonetics, and multichannel character images with ResNet blocks for graphic representations.Source: §1 · p. 2
  • Selective fusion controls each modality’s contribution, while phonetic and graphic encoders are pretrained to predict the correct character from their respective inputs.Source: §1 · p. 2
  • REALISE improves average F1 over SpellGCN by 2.4% at detection-level and 2.6% at correction-level.The comparison is against SpellGCN, a state-of-the-art model using a handcrafted confusion set.Source: §1 · p. 2
  • REALISE performs better on errors absent from handcrafted confusion sets, indicating phonetic and graphic information can better capture easily misused characters.Source: §1 · p. 2

2 Related Work

Earlier Chinese spelling checkers progressed from rules and traditional machine learning to neural and pretrained-language-model approaches. ReaLiSe uses multimodal character information rather than a fixed confusion set, while the related multimodal literature had not previously applied such information to CSC.

2.1 Chinese Spell Checking

  • Chinese spell checking progressed from rule-based methods and traditional machine learning to neural sequence labeling and BERT-based systems.Source: §2.1 · p. 2
  • Handcrafted confusion sets cannot cover all character-similarity relations; ReaLiSe instead uses semantic, phonetic, and graphic encoders with selective fusion to discover similarities.Source: §2.1 · pp. 2–3, Figure 1 · p. 3

2.2 Multimodal Learning

  • Prior multimodal work covered tasks such as sentiment analysis, visual question answering, and machine translation, but this paper presents multimodal information as new to CSC.Source: §2.2 · p. 3

3 The REALISE Model

REALISE uses semantic, phonetic, and graphic encoders to represent Chinese text, then selectively fuses their information before predicting corrected characters. Its acoustic and visual encoders are pretrained with character-recovery and recognition objectives.

3.1 The Semantic Encoder

  • BERT-based Transformer layers turn embedded input tokens into contextualized semantic representations.The final layer output is used as the textual representation.Source: §3.1 · pp. 3–4

3.2 The Phonetic Encoder

  • A hierarchical phonetic encoder represents pinyin at the character level and contextualizes the character representations at the sentence level.The character-level GRU captures pronunciation distinctions; a Transformer sentence-level encoder adds context.Source: §3.2 · p. 4

3.3 The Graphic Encoder

  • A five-block ResNet maps each character image to a visual vector sized for fusion with the semantic representation.Images use three channels corresponding to Gothic Simplified Chinese, Gothic Traditional Chinese, and Small Seal Script fonts.Source: §3.3 · p. 4

3.4 Selective Modality Fusion Module

  • Selective gates regulate how much textual, phonetic, and visual information contributes to each character’s mixed representation.Gate computation uses the three modality representations and the mean semantic output to capture sentence-level semantics.Source: §3.4 · pp. 4–5

3.5 Acoustic and Visual Pretraining

  • Acoustic pretraining recovers character sequences from pinyin, while visual pretraining predicts characters from their images.The pretrained semantic, phonetic, and graphic encoders are then used in final training on CSC data.Source: §3.5 · p. 5

4 Experiments

REALISE is evaluated on the SIGHAN benchmarks, where it outperforms prior state-of-the-art systems. The experiments also examine its components, modality gates, and correction examples.

4.1 Data and Metrics

  • The experiments use SIGHAN training data and Wang271K pseudo data, evaluate SIGHAN13/14/15, and report detection- and correction-level metrics.Detection requires finding all errors in a sentence; correction requires detecting and correcting all of them.Source: §4.1 · pp. 5–6

4.3 Baselines

  • The comparison includes rule-based and traditional machine-learning systems, neural sequence labeling, BERT-based systems, and BERT itself.The described baselines include SpellGCN, which uses predefined confusion sets, and Soft-Masked BERT, which uses a detection model to support correction.Source: §4.3 · p. 7

4.4 Main Results

  • REALISE exceeds BERT in correction-level F1 by 5.2% on SIGHAN13, 3.8% on SIGHAN14, and 4.4% on SIGHAN15.The SIGHAN13 results use the described post-processing, which removes detected and corrected “的”, “地”, and “得” characters before evaluation.Source: §4.4 · p. 7, §4.2 · p. 7
  • REALISE improves average F1 over SpellGCN by 2.4% at detection level and 2.6% at correction level on SIGHAN benchmarks.Table 3 reports REALISE outperforming corresponding baselines across the SIGHAN test sets with p < 0.05.Source: Table 3 · p. 6, §4.4 · p. 7

4.5 Ablation Study

  • Removing either the phonetic or graphic encoder lowers performance at both evaluation levels, while the reduced models still significantly outperform BERT.The ablation table covers five changes: removing either encoder, using one font, removing acoustic and visual pretraining, or replacing selective fusion with summation.Source: §4.5 · pp. 7–8, Table 4 · p. 7

4.6 Analysis of the Selective Modality Fusion Module

  • For example, the selective gates assign larger acoustic and visual values to “埤” when its competing character shares its pronunciation and right radical.A second example shows a similar trend for “带” and “戴,” which share pronunciation.Source: §4.6 · p. 8, Figure 2 · p. 8
  • On SIGHAN15, average gate values rank textual information highest at almost 1.0, acoustic at 0.334, and visual at 0.229.The authors relate the acoustic-over-visual ordering to similar-pronunciation errors being more frequent than similar-shape errors.Source: §4.6 · p. 8

4.7 Case Study

  • Case examples show corrections drawing on character similarities, including phonetic similarity for “流” to “旅” and phonetic plus visual similarity for “跟” to “很”.On SIGHAN15, REALISE corrects 73.5% of erroneous-corrected pairs outside predefined confusion sets, compared with SpellGCN’s 64.6%.Source: §4.7 · pp. 8–9

5 Conclusion

REALISE addresses Chinese spelling errors using textual, acoustic, and visual information, with tailored encoders and selective fusion. On SIGHAN benchmarks, it outperforms textual-only baselines by a large margin.

5 Conclusion

  • REALISE uses textual, acoustic, and visual modalities to detect and correct Chinese spelling errors.It captures information from these modalities with tailored semantic, phonetic, and graphic encoders.Source: §5 · p. 9
  • A selective modality fusion mechanism controls information flow from the three modalities.Source: §5 · p. 9
  • REALISE outperforms textual-only baseline models by a large margin on the SIGHAN benchmarks.The authors conclude that leveraging acoustic and visual information helps Chinese spell checking.Source: §5 · p. 9

A.1 Ablation

The ablation study tests the model’s encoders, graphic input diversity, pretraining objectives, and fusion mechanism. Removing any component consistently lowers performance, most apparently on SIGHAN14.

A.1 Ablation

  • The ablations remove the phonetic or graphic encoder, or restrict the graphic encoder to one Gothic font.Source: §A.1 · p. 12
  • Other ablations remove acoustic and visual pretraining objectives or replace selective fusion with simple summation.Source: §A.1 · p. 12
  • Removing any model component consistently lowers REALISE’s performance, with the largest apparent drops on SIGHAN14.Source: §A.1 · p. 12

A.2 Selective Modality Fusion Visualization

The visualization relates modality gate values to the type of character similarity. Acoustic gate values tend to be larger for phonetic similarity, while visual gate values tend to be larger for graphic similarity.

A.2 Selective Modality Fusion Visualization

  • The section presents additional examples in Figure 3.Source: §A.2 · p. 12
  • For phonetically similar errors, acoustic gate values tend to be larger.Source: §A.2 · p. 12
  • For graphically similar errors, visual gate values tend to be larger.Source: §A.2 · p. 12

Related papers

Loading 2105.12306v1…