Source-linked AI summary
UER: An Open-Source Toolkit for Pre-training Models
Zhe Zhao, Hui Chen, Jinbin Zhang, Xin Zhao, Tao Liu, Wei Lu, Xi Chen, Haotang Deng, Qi Ju, Xiaoyong Du
TL;DR
Because pre-training models differ and no single model suits every case, an efficient framework for deploying varied models is needed. UER offers modules for assembling existing or customized pre-training models and a model zoo spanning corpora, encoders, and objectives. The reported models achieve state-of-the-art or competitive downstream results, with competitive results also reported at efficient running speed.
Problem
Pre-training models differ across corpora, encoders, and objectives, and no single model works best in all cases.
Source: §1 · p. 1Method
UER is a modular toolkit whose components can be assembled to reproduce existing pre-training models or develop customized ones.
Source: §1 · p. 2Results
UER-based models achieve significant improvements over BERT or competitive results with efficient training speed.
Source: §5 · p. 6Takeaways & Limitations
UER's model zoo lets users select pre-trained models across corpora, encoders, and targets for downstream datasets.
Source: §1 · p. 2The analysis of target and encoder influence uses sentiment-analysis datasets as examples.
Source: §4.2 · p. 5
Abstract
from arXiv · showhide
Existing works, including ELMO and BERT, have revealed the importance of pre-training for NLP tasks. While there does not exist a single pre-training model that works best in all cases, it is of necessity to develop a framework that is able to deploy various pre-training models efficiently. For this purpose, we propose an assemble-on-demand pre-training toolkit, namely Universal Encoder Representations (UER). UER is loosely coupled, and encapsulated with rich modules. By assembling modules on demand, users can either reproduce a state-of-the-art pre-training model or develop a pre-training model that remains unexplored. With UER, we have built a model zoo, which contains pre-trained models based on different corpora, encoders, and targets (objectives). With proper pre-trained models, we could achieve new state-of-the-art results on a range of downstream datasets.
1 Introduction
Pre-training models vary in their encoders, objectives, corpora, and fine-tuning strategies, and no single model works best across all scenarios. UER provides modular components for assembling existing or customized models, and its Chinese model zoo yields state-of-the-art or competitive, efficient downstream results.
1 Introduction
- Pre-training models differ across encoders, objectives, corpora, and fine-tuning strategies, while no single model works best in every scenario.Different downstream tasks and computational constraints can call for different models.Source: §1 · pp. 1–2
- UER combines subencoder, encoder, target, and fine-tuning components to reproduce models such as BERT or add customized modules.Its interfaces are designed to allow module assembly with few restrictions.Source: §1 · p. 2, Figure 1 · p. 2
- UER's Chinese model zoo spans different corpora, encoders, and targets; selected models achieve new state-of-the-art results or competitive results with efficient running speed.The paper reports using Google BERT as the baseline.Source: §1 · p. 2
- UER is built on PyTorch, supports distributed training, and is publicly available with documentation.The toolkit and model zoo are available through the project's GitHub repository.Source: §1 · p. 2
2 Related Work
Pre-training initializes entire networks from large corpora before fine-tuning them for downstream tasks; BERT is influential but computationally expensive. UER extends the toolkit approach to support flexible construction of varied pre-training models.
2.1 Pre-training for deep neural networks
- BERT pre-trains deep Transformer networks with masked-language and sentence-prediction objectives, but requires expensive computational resources.Google's released pre-trained models can instead be fine-tuned directly.Source: §2.1 · pp. 2–3
2.2 NLP toolkits
- Existing NLP toolkits implement model classes across tasks such as translation, classification, sequence labeling, and word embedding.Examples include OpenNMT, fairseq, NCRF++, and Hyperwords.Source: §2.2 · p. 3
- UER applies the toolkit approach to pre-training, providing flexibility to build models with different properties and facilitate reproducing or refining them.The related work identifies GPT, ULMFiT, and BERT as influential pre-training works.Source: §2.2 · p. 3
3 Architecture
UER organizes pre-training into modular subencoder, encoder, and target components, followed by downstream fine-tuning. It offers varied feature extractors and objectives, supports alternative fine-tuning stages, and allows users to assemble or customize models through its interfaces.
3 Architecture
- UER's architecture organizes pre-training modules and presents case studies to show their use.The framework's modular design is intended to facilitate use of pre-training models.Source: §3 · p. 3
3.1 Subencoder
- The subencoder learns word vectors from subword features, using character features for English and radical and pinyin features for Chinese.UER implements RNN and CNN subencoders and uses mean or max pooling to produce fixed-length word vectors.Source: §3.1 · p. 3
3.2 Encoder
- UER's encoder modules include LSTM, GRU, CNN, GatedCNN, AttentionNN, and Transformer, and can be stacked or combined into custom networks.Transformer modules can be used to implement models such as GPT and BERT.Source: §3.2 · p. 3
3.3 Target (objective)
- UER supports word-level objectives including language modeling, masked language modeling, and autoencoding, as well as sentence-level next-sentence prediction.Masked language modeling uses both left and right context; language modeling uses only left context. Sentence prediction avoids sequential word decoding and a vocabulary-wide softmax.Source: §3.3 · pp. 3–4
- UER also supports supervised objectives such as natural-language-inference classification and neural machine translation; objectives can be combined with different weights.The paper notes that supervised tasks can provide knowledge unavailable from raw corpora, and that some objectives may be complementary.Source: §3.3 · pp. 3–4
3.4 Fine-tuning
- UER fine-tunes the whole model on downstream tasks and supports an additional unsupervised pre-training stage on the downstream dataset.That intermediate stage is described as helping models become familiar with downstream data distributions.Source: §3.4 · p. 4
3.5 Case Studies
- Case studies show that BERT, GPT, Quick-thoughts, and InferSent can be constructed by specifying combinations of embedding, encoder, and target modules.Users can select modules through options without code changes or integrate customized modules into UER.Source: §3.5 · p. 4
4 Experiments
UER is evaluated on English and Chinese benchmarks, where its BERT implementation matches public implementations and selected pre-training models improve downstream performance. Amazon-review pre-training with CLS targets and an LSTM encoder can perform well against the BERT baseline, subject to the section’s sentiment-analysis scope.
4 Experiments
- Evaluation covers sentence and sentence-pair classification, sequence labeling, and document-based QA, using English and Chinese BERT baselines.English tests use GLUE development benchmarks; Chinese tests span five dataset types.Source: §4 · p. 4
- Proper encoders and targets yield significant improvements on downstream datasets with models from UER’s zoo.The experiments apply zoo models to different downstream datasets.Source: §4 · p. 5
4.1 Reproducibility
- UER’s BERT implementation matches HuggingFace’s and ERNIE’s implementations on the reported English and Chinese benchmarks.Tables 2 and 3 compare the implementations; the reported results are available from their project repositories, and UER results can be reproduced with its scripts.Source: §4.1 · p. 5, Table 2 · p. 5, Table 3 · p. 5
4.2 Influence of targets and encoders
- Amazon-review pre-training improves sentiment results over the Chinese Wikipedia BERT baseline, and CLS achieves the best results in most cases.Reviews’ user ratings support using a CLS target; NSP is removed because reviews are split into sentence-level parts.Source: §4.2 · p. 5, Table 4 · p. 5
- A 2-layer LSTM encoder achieves comparable or better results than BERT when suitable corpora and targets are selected.The comparison replaces BERT’s 12-layer Transformer encoder with an LSTM to improve efficiency.Source: §4.2 · p. 5, Table 5 · p. 5
- The target and encoder analysis uses sentiment-analysis datasets as examples because of space constraints; more tasks and models are discussed in UER’s GitHub project.The section’s scope is limited to three Chinese sentiment datasets.Source: §4.2 · pp. 5–6
5 Conclusion
UER is an open-source toolkit for pre-training and downstream fine-tuning, with models spanning different corpora, encoders, and targets. The paper reports improvements over BERT with suitable models, or competitive results at efficient training speed.
5 Conclusion
- UER is an open-source toolkit for pre-training and fine-tuning, with public models spanning corpora, encoders, and targets.The paper reports that suitable pre-trained models can improve over BERT or deliver competitive results with efficient training speed.Source: §5 · p. 6