Source-linked AI summary
CharBERT: Character-aware Pre-trained Language Model
Wentao Ma, Yiming Cui, Chenglei Si, Ting Liu, Shijin Wang, Guoping Hu
TL;DR
Subword-based PLMs can produce incomplete and fragile word representations, motivating CharBERT’s combination of character and subword information with unsupervised character learning. The paper reports improved performance and robustness across its evaluations.
Problem
BPE-based PLMs can miss whole-word and fine-grained character information, and minor typos can drastically change subword tokens.
Source: §1 · p. 1Method
CharBERT builds contextual word representations from sequential character representations, fuses them with subword representations, and uses noisy-character pre-training for unsupervised character learning.
Source: §1 · p. 2, §3.3 · p. 4Results
CharBERT significantly improves performance over strong BERT and RoBERTa baselines on eight benchmarks and reports large-margin robustness gains on three character-attack test sets.
Source: §1 · p. 2Takeaways & Limitations
The conclusion reports that CharBERT improves both performance and robustness of pre-trained language models.
Source: §6 · p. 10
Abstract
from arXiv · showhide
Most pre-trained language models (PLMs) construct word representations at subword level with Byte-Pair Encoding (BPE) or its variations, by which OOV (out-of-vocab) words are almost avoidable. However, those methods split a word into subword units and make the representation incomplete and fragile. In this paper, we propose a character-aware pre-trained language model named CharBERT improving on the previous methods (such as BERT, RoBERTa) to tackle these problems. We first construct the contextual word embedding for each token from the sequential character representations, then fuse the representations of characters and the subword representations by a novel heterogeneous interaction module. We also propose a new pre-training task named NLM (Noisy LM) for unsupervised character representation learning. We evaluate our method on question answering, sequence labeling, and text classification tasks, both on the original datasets and adversarial misspelling test sets. The experimental results show that our method can significantly improve the performance and robustness of PLMs simultaneously. Pretrained models, evaluation sets, and code are available at https://github.com/wtma/CharBERT
1 Introduction
BPE-based PLMs avoid most OOV words but can omit whole-word and fine-grained character information, while small typos can change subword segmentation. CharBERT adds character representations and reports improved benchmark performance and robustness.
1 Introduction
- BPE can leave word representations incomplete and fragile: minor typos may change both the subwords and their number.The paper illustrates this with “backhand”: removing “k” changes its subword segmentation.Source: §1 · p. 1, Figure 1 · p. 2
- Nearly 28% of CoNLL-2003 NER development-set nouns split into multiple BERT subwords; removing a character changes subwords for about 78% and subword counts for 77%.These figures describe random character removal from the noun words.Source: §1 · p. 1
- CharBERT builds contextual word representations from sequential character representations and fuses them with subword representations after each transformer block.Its character encoder uses context string embeddings rather than a traditional CNN; the proposed method can also use backbones such as RoBERTa.Source: §1 · p. 2
- The proposed unsupervised character-learning task injects character noise and trains the model to restore the original word.The paper names this pre-training task NLM.Source: §1 · p. 2
- CharBERT improves performance over strong BERT and RoBERTa baselines on eight benchmarks and reports large-margin robustness gains on three character-attack test sets.The attacks cover three task types.Source: §1 · p. 2
2 Related Work
Related work covers the development of pre-trained language models, character-based representations, adversarial attacks, and methods for fusing heterogeneous representations. The paper positions its two-step character–subword fusion as distinct from earlier fusion modules.
2 Related Work
- Transformer-based PLMs including BERT and later models use large-scale pre-training, while BPE splits words into subword units.The passage also describes earlier RNN-based models and GPT’s transformer-decoder approach.Source: §2 · p. 2
- Character language models perform competitively on various NLP tasks, including neural machine translation and sequence labeling; character embeddings also construct contextual word representations, with Peters et al. (2018) reporting significant improvement.Source: §2 · p. 3
- Prior work studies adversarial attacks on reading comprehension and character-level machine translation, while spelling correction has been proposed as a defense.The cited attacks include black-box, white-box, and character-level approaches.Source: §2 · p. 3
- Earlier heterogeneous-fusion modules appear in reading comprehension and PLMs; this work instead proposes two-step interactive fusion of character and subword representations.The authors say the design could be extended to integrate information such as diacritics or external knowledge.Source: §2 · p. 3
3 Methodology
CharBERT combines BPE subword and character channels, using character representations and heterogeneous interaction to enrich the model’s representations. Its methodology also includes noisy character pretraining and task-specific fine-tuning.
3.2 Model Architecture
- CharBERT uses a dual-channel architecture, with separate character and subword representations that are fused and split by heterogeneous interaction.The module is applied after each transformer layer; fusion lets the channels enrich each other, while splitting retains their distinct features.Source: §3.2 · p. 3, §3.4 · pp. 4–5
3.3 Character Encoder
- Character representations come from a bi-GRU over the full input character sequence, with each token embedding formed by concatenating its first and last character hidden states.Processing the full sequence captures character information within and among subwords.Source: §3.3 · p. 4
3.4 Heterogeneous Interaction
- The heterogeneous interaction module fuses the channel representations with a CNN, then uses a fully connected layer and residual connections to produce updated channel representations.Layer normalization follows the residual connection.Source: §3.4 · pp. 4–5
3.5 Unsupervised Character Pre-training
- Noisy language modeling (NLM) trains the character channel to recover original whole words after internal characters are dropped, added, or swapped.The prediction space is a word-level vocabulary; MLM is also used for the token channel, with tokens masked or replaced without character noise.Source: §3.5 · p. 5
3.6 Fine-tuning for Specific Tasks
- For token-level tasks, fine-tuning concatenates the final embeddings from both channels; for sequence-level classification, it averages the concatenated last-layer embeddings.Table 1 reports BERT-base results on question-answering and text-classification development sets, using task-specific metrics.Source: §3.6 · pp. 5–6, Table 1 · p. 6
4 Experiments
The experiments evaluate CharBERT on question answering, text classification, and sequence labeling, alongside character-attack tests and ablations. Across the reported tasks, the authors compare it with BERT and RoBERTa baselines and find stronger sequence-labeling results and greater robustness to misspellings.
4.1 Experimental Setup
- The experiments use BERT-base and RoBERTa-base as main baselines, with 5M additional character-channel parameters for each model.CharBERT is pretrained for 320K steps on English Wikipedia; the setup lowers MLM masking to 10% and changes 15% of input words for NLM.Source: §4.1 · p. 6
4.2 Results on Question Answering (SQuAD)
- CharBERT and CharBERTRoBERTa outperform the baseline pre-trained models on SQuAD results except for RoBERTa on SQuAD 1.1.The SQuAD 2.0 fine-tuning method concatenates character- and token-channel outputs to predict answer-span start and end positions.Source: §4.2 · pp. 6–7
4.3 Results on Text Classification
- CharBERT significantly outperforms BERT on all four text-classification tasks, while gains over the stronger RoBERTa baseline are much weaker.The authors report weaker classification improvements than for question answering and sequence labeling.Source: §4.3 · p. 7
4.4 Results on Sequence Labeling
- CharBERT and CharBERTRoBERTa significantly outperform BERT and RoBERTa on sequence labeling and set new state-of-the-art results on POS tagging.The reported significance threshold is p-value ≤0.05.Source: §4.4 · p. 7, Table 3 · p. 8
4.5 Robustness Evaluation
- On character-attack sets, BERT loses more than 30% performance, while CharBERT has the smallest drop among the compared baselines and improves on original test sets.The evaluation covers SQuAD 2.0, CoNLL-2003 NER, and QNLI; Table 2 reports accuracy for QNLI and F1 for NER and SQuAD 2.0.Source: Table 2 · p. 7, §4.5 · p. 8
4.6 Ablation Study
- Removing the GRU or heterogeneous interaction reduces performance across tasks; removing NLM preserves similar clean-set results but worsens QNLI-Att performance.CharBERT without NLM still performs better than BERT on the attack set; AdvBERT is the matched-pretraining baseline without CharBERT’s three modules.Source: §4.6 · pp. 8–9, Table 4 · p. 8
5 Analysis
The analyses examine whether subword splitting limits NER representations, how embeddings respond to character noise, and whether character features enrich contextual embeddings.
5.1 Word vs. Subword
- 0.68% vs. 0.29%: CharBERT's improvement was larger for Subword than Word examples, although Subword examples were 17.8% of words and 45.3% of named entities.The authors attribute the improvement on words split across subwords to the GRU layer capturing information among those subwords.Source: §5.1 · p. 9
5.2 Robustness Analysis
- The sensitivity analysis measures how contextual word embeddings change between original text and text with character attacks.For words split into multiple subwords, the experiment uses the first subword's hidden representation as the word embedding.Source: §5.2 · pp. 9–10
- 0.0612 (BERT), 0.0407 (AdvBERT), and 0.0986 (CharBERT): the sensitivity scores showed no significant correlation with the models' reported robustness ordering.The authors suggest CharBERT's NLM approach to noisy words may improve robustness differently from AdvBERT's treatment of all words during pretraining.Source: §5.2 · p. 10
5.3 Feature-based Comparison
- Concatenating CharBERT's token- and character-channel embeddings achieved the best feature-based score; token-channel performance was similar to BERT's, while character-only performance was worse.The embeddings were used as features for a randomly initialized two-layer 768-dimensional Bi-LSTM, without fine-tuning BERT or CharBERT.Source: §5.3 · p. 10
6 Conclusion
The paper concludes that CharBERT addresses incomplete modeling and robustness limitations, with experiments reporting improvements in both performance and robustness. Future work proposes extending it to other languages and noise types.
6 Conclusion
- CharBERT improves both performance and robustness of pretrained language models, according to the reported experiments.The authors frame the approach as addressing incomplete modeling and lack of robustness.Source: §6 · p. 10
- CharBERT combines sequential GRU character representations with subword and character channels, heterogeneous interaction, and NLM for unsupervised character representation learning.Source: §6 · p. 10
- The authors propose extending CharBERT to other languages and to word-level and sentence-level noise.They specifically identify morphologically rich languages such as Arabic as a future direction.Source: §6 · p. 10