Source-linked AI summary
CharacterBERT: Reconciling ELMo and BERT for Word-Level Open-Vocabulary Representations From Characters
Hicham El Boukkouri, Olivier Ferret, Thomas Lavergne, Hiroshi Noji, Pierre Zweigenbaum, Junichi Tsujii
TL;DR
General-domain wordpiece vocabularies may not suit specialized domains, while subword tokenization moves representations away from the word level. CharacterBERT replaces BERT’s wordpiece system with a Character-CNN that represents whole words from characters, and it improves specialized medical performance while offering word-level open-vocabulary representations and greater robustness to misspellings. The study is limited to English and medical domains, and CharacterBERT is slower to pre-train.
Problem
General-domain wordpiece vocabularies may be unsuitable for specialized domains, while wordpiece tokenization shifts representations from the word level to subwords.
Method
CharacterBERT replaces BERT’s wordpiece system with a Character-CNN that constructs token representations from characters while retaining BERT’s Transformer architecture.
Results
CharacterBERT globally outperforms BERT when specialized for the medical domain and is more robust to misspellings.
Takeaways & Limitations
CharacterBERT provides word-level open-vocabulary representations while improving specialized medical performance and robustness to misspellings.
Takeaways & Limitations
The work is limited to English and medical domains, and CharacterBERT is slower to pre-train than BERT.
Abstract
from arXiv · showhide
Due to the compelling improvements brought by BERT, many recent representation models adopted the Transformer architecture as their main building block, consequently inheriting the wordpiece tokenization system despite it not being intrinsically linked to the notion of Transformers. While this system is thought to achieve a good balance between the flexibility of characters and the efficiency of full words, using predefined wordpiece vocabularies from the general domain is not always suitable, especially when building models for specialized domains (e.g., the medical domain). Moreover, adopting a wordpiece tokenization shifts the focus from the word level to the subword level, making the models conceptually more complex and arguably less convenient in practice. For these reasons, we propose CharacterBERT, a new variant of BERT that drops the wordpiece system altogether and uses a Character-CNN module instead to represent entire words by consulting their characters. We show that this new model improves the performance of BERT on a variety of medical domain tasks while at the same time producing robust, word-level and open-vocabulary representations.
1 Introduction
CharacterBERT addresses whether general-domain wordpiece vocabularies suit specialized domains by replacing BERT’s wordpiece system with character-based word representations. The model targets word-level, open-vocabulary representations and reports improved medical-task performance and robustness to noise and misspellings.
- General-domain wordpiece vocabularies may be unsuitable for specialized domains, motivating evaluation of their domain fit.
- CharacterBERT replaces BERT’s predefined wordpiece vocabulary with an ELMo Character-CNN while retaining the rest of BERT’s architecture.
- CharacterBERT produces word-level contextualized representations without requiring a wordpiece vocabulary.
- CharacterBERT outperforms BERT on multiple specialized medical tasks without requiring a wordpiece vocabulary.
- The authors report improved robustness to noise and misspellings in favor of CharacterBERT.
- The study is limited to English and medical domains, leaving other languages and specialized domains for future work.
2 General-Domain Wordpieces in Specialized Domains
The authors compare general- and medical-domain vocabularies on medical text and find that vocabulary choice affects both tokenization quantity and wordpiece meaningfulness. These findings motivate a wordpiece-free model for specialized applications.
- The experiment processes one million medical-corpus tokens with medical and original BERT vocabularies to compare their tokenization.
- The medical vocabulary produces fewer wordpieces than the general vocabulary at both occurrence and type levels.
- The specialized vocabulary produces more meaningful units for specific medical terms, including “choledoch” and “olithiasis.”
- The analyses indicate that vocabulary choice affects tokenization quality and may induce downstream representation-model biases.
3 CharacterBERT
CharacterBERT constructs one token representation from characters instead of splitting tokens into wordpieces, then feeds these representations through BERT’s Transformer layers. Its pre-training predicts whole words rather than individual wordpieces.
- CharacterBERT uses an ELMo Character-CNN to produce one representation for each token, whereas BERT independently embeds multiple wordpieces.
- The Character-CNN converts each token into characters, embeds them, applies multiple one-dimensional CNNs with max pooling, and concatenates their outputs.
- Two Highway layers transform the CNN representation before projecting it to a size compatible with BERT’s 768-dimensional wordpiece representations.
- CharacterBERT adds its character-based token representation to position and segment embeddings before the Transformer layers.
- Because tokens are not split into wordpieces, each input token receives a single final contextual representation.
- CharacterBERT retains BERT’s MLM and NSP pre-training tasks but predicts entire words rather than individual wordpieces in MLM.
4 Experiments
The experiments compare BERT and CharacterBERT under matched conditions across five medical tasks, using general- and medical-domain pre-training variants. Evaluation averages performance across ten random seeds and also constructs ensembles from those seeds.
- Experimental setup: CharacterBERT and BERT are trained in the same conditions to isolate the effect of replacing wordpieces with a Character-CNN.Both models use the base-uncased BERT configuration, while CharacterBERT represents tokens from characters and produces word-level representations.
- Model settings: CharacterBERT uses a Character-CNN instead of BERT’s wordpiece embedding layer while retaining the rest of the Transformer architecture.The character representation uses byte-level token encoding and seven one-dimensional CNNs with filters spanning character n-grams.
- Pre-trained models: Four pre-trained models simulate general-domain pre-training followed by optional medical-domain retraining: BERTgeneral, CharacterBERTgeneral, BERTmedical, and CharacterBERTmedical.The medical variants are retrained on a corpus built from MIMIC-III and PMC OA abstracts.
- Evaluation tasks: The study evaluates five medical tasks covering entity recognition, natural language inference, relation classification, and sentence similarity.The tasks include i2b2/VA 2010, MEDNLI, ChemProt, DDI, and ClinicalSTS.
- Evaluation protocol: Each model-task-seed combination is fine-tuned for 15 epochs, selecting the best validation checkpoint before test evaluation.Ten random seeds yield final performance reported as mean ± std, and majority-vote ensembles are built from the seed-specific models.
5 Results and Discussion
CharacterBERT has a substantial pre-training speed disadvantage but a smaller fine-tuning disadvantage, and it can be faster than BERT for a task with many domain-specific terms. The paper attributes the pre-training gap mainly to the masked-language-modeling output setup.
- 5.1.1 Pre-training: 55 hours versus 26.5 hours: CharacterBERT takes about twice as long as BERT to complete pre-training despite similar parameter counts.The slower training is partly attributed to the more complex Character-CNN and mainly to the inability to share the wordpiece embedding matrix with the MLM output layer.
- 5.1.2 Fine-tuning: 19% slower on average instead of 108%: CharacterBERT is much less disadvantaged than BERT during fine-tuning.The comparison concerns fine-tuning speed rather than pre-training speed.
- 5.1.2 Fine-tuning: 14% faster: CharacterBERT outperforms BERT in DDI fine-tuning speed.The paper relates this case to domain-specific terms being split into multiple wordpieces by BERT, while CharacterBERT keeps word-level input size stable.
- 5.1.2 Fine-tuning: CharacterBERT is slightly faster than BERT at inference time.The supplied discussion reports this inference-time comparison after describing the fine-tuning results.
5.2 Reproducing Vanilla Models
The reproduced baselines provide comparisons among original BERT, retrained general-domain BERT, and medical-domain models, while the evaluation also distinguishes single-model results from ensembles. Ensembles improve performance and reduce variance relative to single models.
- Baseline reproduction: BERTgeneral performs slightly worse than original BERT despite using the same architecture.The paper attributes the small difference to different general-domain corpora or pre-training parameters.
- Baseline reproduction: BERTmedical performs at the same level as BlueBERT and sometimes outperforms it.BlueBERT is pre-trained on MIMIC-III and PubMed abstracts, whereas the paper’s medical models use PMC OA abstracts and MIMIC-III.
- Ensembles and model selection: Ensembles clearly improve over single models across the displayed medical-task comparisons.The ensembles use a naive majority-voting strategy, with continuous ClinicalSTS scores handled by averaging predictions.
- Ensembles and model selection: Ensemble results have substantially lower variance than single-model results, making them more reliable for comparisons.The figure reports single-model results as blue, ensemble results as orange, and best-validation-seed results as red.
5.4 BERT vs. CharacterBERT: How Significant Is the Difference?
CharacterBERT often improves over BERT on medical tasks, although ClinicalSTS is an exception with high variance. Statistical testing and noise experiments further examine whether the observed differences are reliable and whether character-based representations improve robustness to misspellings.
- 5.4 BERT vs. CharacterBERT: ≈1.5 points on ChemProt, ≈2 points on DDI, and ≈0.5 points on MEDNLI and i2b2: CharacterBERTmedical improves over BERTmedical ensembles.CharacterBERTmedical performs worse than BERT on ClinicalSTS, where the dataset is small and variance is high.
- 5.4 BERT vs. CharacterBERT: Almost Stochastic Order tests compute a task-specific distance ϵ from significant stochastic superiority using the ten single-model scores for each model pair.ϵ equals 0 when model A is at least as good in the tested order, 1 when model B is, and 0.5 when no order can be established.
- 5.5 Robustness to Noise and Misspellings: 40% test-set noise: CharacterBERT is ≈5 F1 higher than BERT, compared with an original difference below 1 F1.Noise consists of character removals, additions, replacements, or swaps; when noise affects all splits, both models become more robust but CharacterBERT remains ahead.
5.6 Discussion and Future Work
CharacterBERT generally matches or improves on BERT, particularly in specialized versions, and is more robust to misspellings. Its main trade-off is slower pre-training, despite comparable fine-tuning and slightly faster inference.
- CharacterBERT generally performs at the same level as or better than BERT, especially in specialized versions.The authors report that this pattern is further validated by ASO tests.
- CharacterBERT remains more robust to misspellings while producing word-level open-vocabulary representations.The authors describe this robustness as desirable because BERT is sensitive to misspellings.
- CharacterBERT is slower to pre-train, but it is not as slow to fine-tune and is slightly faster at inference time.
6 Conclusion
CharacterBERT replaces specialized BERT models’ general-domain wordpiece vocabulary with a Character-CNN that produces word-level open-vocabulary representations. In medical specialization, it globally outperforms BERT while being more robust to misspellings.
- CharacterBERT replaces BERT’s general-domain wordpiece system with a Character-CNN that consults token characters.The module produces word-level open-vocabulary representations.
- CharacterBERT globally outperforms BERT when specialized for the medical domain.
- CharacterBERT is more robust to misspellings than BERT.
A.1 Detailed Test Scores
The detailed test scores report model performance using first, median, and third quartiles. Figure 8 compares pre-trained models fine-tuned on five medical tasks, including general-domain BERT and medical BlueBERT baselines.
- Detailed model scores are reported as first, second (median), and third quartiles to show score distributions.
- Figure 8 compares pre-trained models fine-tuned on five different medical tasks.
- The comparison includes base-uncased BERT and BlueBERT, a medical BERT retrained on MIMIC-III and PubMed abstracts.