Source-linked AI summary
Trankit: A Light-Weight Transformer-based Toolkit for Multilingual Natural Language Processing
Minh Van Nguyen, Viet Dac Lai, Amir Pouran Ben Veyseh, Thien Huu Nguyen
TL;DR
Multilingual fundamental NLP pipelines must support raw text and many languages without sacrificing task performance or efficiency. Trankit uses shared multilingual transformer components with adapters to provide such a toolkit, achieving stronger results on several core tasks while remaining competitive on others. Its evaluation covers 90 treebanks across 56 languages, with a stated limitation that 10 unsupported-language treebanks are omitted.
Problem
Existing multilingual toolkits have limitations in performance, raw-text processing, external-tokenizer dependence, and efficient multilingual model deployment.
Method
Trankit uses a shared multilingual pretrained transformer with adapter and task-specific weights in a multilingual pipeline for fundamental NLP tasks.
Results
Across 90 treebanks, Trankit outperforms Stanza particularly on sentence segmentation, POS tagging, morphological tagging, and dependency parsing while remaining competitive on tokenization, multi-word expansion, and lemmatization.
Takeaways & Limitations
Trankit provides a fast, memory-efficient multilingual toolkit for raw-text fundamental NLP processing across 56 languages.
Takeaways & Limitations
The evaluation omits 10 treebanks whose languages are unsupported by XLM-Roberta.
Abstract
from arXiv · showhide
We introduce Trankit, a light-weight Transformer-based Toolkit for multilingual Natural Language Processing (NLP). It provides a trainable pipeline for fundamental NLP tasks over 100 languages, and 90 pretrained pipelines for 56 languages. Built on a state-of-the-art pretrained language model, Trankit significantly outperforms prior multilingual NLP pipelines over sentence segmentation, part-of-speech tagging, morphological feature tagging, and dependency parsing while maintaining competitive performance for tokenization, multi-word token expansion, and lemmatization over 90 Universal Dependencies treebanks. Despite the use of a large pretrained transformer, our toolkit is still efficient in memory usage and speed. This is achieved by our novel plug-and-play mechanism with Adapters where a multilingual pretrained transformer is shared across pipelines for different languages. Our toolkit along with pretrained models and code are publicly available at: https://github.com/nlp-uoregon/trankit. A demo website for our toolkit is also available at: http://nlp.uoregon.edu/trankit. Finally, we create a demo video for Trankit at: https://youtu.be/q0KGP3zGjGc.
1 Introduction
Trankit addresses limitations in multilingual fundamental NLP toolkits with a transformer-based pipeline that processes raw text across many languages. It combines wordpiece-based splitting, shared multilingual transformer components, and adapters to improve performance while reducing memory and training costs.
- Motivation: Existing multilingual toolkits trade off speed, performance, raw-text processing, or reliance on external tokenizers.spaCy sacrifices performance for speed; UDify and Flair require external tokenizers; CoreNLP lacks state-of-the-art performance; UDPipe and Stanza use word embeddings.
- Contribution: Trankit supports raw text for fundamental NLP tasks across 56 languages with 90 pretrained pipelines on Universal Dependencies v2.5 treebanks.
- Contribution: Trankit improves sentence segmentation, POS tagging, morphological tagging, and dependency parsing while remaining competitive for tokenization, multi-word expansion, and lemmatization.
- Novelty: Wordpiece-based token and sentence splitting exploits contextual information instead of character-level boundaries.The approach is designed to benefit many languages.
- Novelty: Trankit correctly keeps a sentence containing “Argghhh!” intact, whereas character-based splitters in Stanza and UDPipe divide it at the exclamation mark.The authors describe this as the first successful wordpiece-based token and sentence splitter for 56 languages.
- Architecture: Adapters activate language- and task-specific weights while sharing a multilingual pretrained transformer across pipeline components.The plug-and-play mechanism reduces memory use and substantially reduces training time.
2 Related Work
Prior multilingual transformer systems often target selected tasks or depend on external resources for tokenization and sentence segmentation. Trankit extends transformer-based multilingual processing to raw text across many languages within one toolkit.
- Prior Work: Earlier transformer-based work addressed individual tasks such as segmentation, POS tagging, morphological tagging, or dependency parsing in limited language settings.
- Prior Work: Multilingual transformer systems trained on combined multilingual data can be suboptimal.The paper discusses multilingual training as suboptimal in its ablation analysis.
- Gap: Existing multilingual transformer systems still rely on external resources for tokenization and sentence segmentation, preventing raw-text input.
- Contribution: Trankit is presented as the first multilingual transformer-based toolkit to load models for many languages simultaneously and process raw text.
3 Design and Architecture
Trankit uses adapters around a shared multilingual transformer and combines specialized components to perform tokenization, sentence splitting, tagging, parsing, lemmatization, and named entity recognition. Its architecture shares representations while keeping task- and language-specific processing modular.
- Multilingual Encoder with Adapters: Adapters place lightweight projection layers inside pretrained transformer layers while retaining residual connections.During training, the pretrained transformer weights remain fixed; adapter and task-specific weights are updated.
- Multilingual Encoder with Adapters: A shared multilingual encoder converts raw text into wordpiece representations for each language and pipeline component.SentencePiece produces wordpieces, and inputs longer than 512 tokens are divided into consecutive chunks.
- Joint Token and Sentence Splitter: The joint token and sentence splitter predicts word, multi-word-token, and sentence boundaries from wordpiece representations.A feed-forward network with softmax classifies wordpiece endpoints, whose predictions are aggregated into boundaries.
- Other Components: Separate character-based sequence-to-sequence models handle multi-word token expansion and lemmatization, while a transformer-based BIOES-CRF model recognizes entities.The lemmatizer receives sentences and predicted UPOS tags; the entity recognizer assigns BIOES tags to token sequences.
- Joint Model for POS Tagging, Morphological Tagging and Dependency Parsing: One sentence-level model jointly performs POS tagging, morphological feature tagging, and dependency parsing.Word representations are formed by averaging wordpiece representations; tagging uses softmax, while parsing uses Deep Biaffine Attention and Chu-Liu/Edmonds decoding.
4 Usage
Trankit provides installation, pretrained-pipeline initialization, multilingual processing, raw or pretokenized input handling, and trainable pipelines. Its usage interface supports processing outputs through Python dictionaries and includes a multilingual pipeline workflow.
- Installation: Trankit is distributed as a Python package through PyPI and provides documentation for installation and use.Users install the toolkit with pip.
- Pipeline Initialization: A pretrained English pipeline can be initialized for GPU execution with a specified model cache directory.Existing pretrained models in the cache are not downloaded again.
- Multilingual Usage: The multilingual pipeline initializes Trankit to process inputs written in different languages.Figure 3 presents the multilingual initialization procedure.
- Basic Functions: Trankit accepts raw or pretokenized strings at sentence and document levels and organizes results in hierarchical Python dictionaries.Figure 4 illustrates a function that performs all supported tasks on an input text.
- Training your own Pipelines: The TPipeline class supports training custom pipelines for 100 languages.This capability is provided as part of Trankit's trainable pipeline interface.
- Demo Website: Trankit also provides a demo website supporting 90 pretrained pipelines.The website interface is illustrated in Figure 7.
5 System Evaluation
Trankit is evaluated against multilingual NLP toolkits on Universal Dependencies, named entity recognition, efficiency, and adapter-based design. It shows strong performance across fundamental tasks while maintaining competitive resource usage, and adapter-based training outperforms alternative multilingual strategies.
- Datasets & Hyper-parameters: Trankit is evaluated on the canonical splits of 90 Universal Dependencies v2.5 treebanks and 11 public NER datasets.The evaluation follows Stanza’s setup and uses official or inherited canonical data splits.
- Universal Dependencies performance: 16.16% improvement in Arabic sentence segmentation and 12.31% UAS and 12.72% LAS improvements in Chinese demonstrate substantial gains on selected languages.English also shows a 7.22% sentence-segmentation improvement and 3.92% UAS and 4.37% LAS improvements.
- Universal Dependencies performance: Over 90 treebanks, Trankit improves over Stanza by +3.24% for sentence segmentation, +1.44% UPOS, +1.46% morphological tagging, and +4.0% UAS.It also improves XPOS by +1.55% and LAS by +5.01%, while remaining competitive on tokenization, multi-word expansion, and lemmatization.
- Speed and Memory Usage: Trankit loads all 90 pretrained pipelines for 56 languages using 4.9GB, while Stanza packages consume hundreds of megabytes for each language.Trankit’s language-specific model packages, excluding the multilingual transformer, take only dozens of megabytes.
- Ablation Study: Multilingual and No-adapters alternatives are significantly worse than adapter-based Trankit on macro-averaged F1 across nine treebanks.The paper attributes the first gap to unbalanced treebank sizes and the second to limitations from fixing the pretrained transformer.
6 Conclusion and Future Work
Trankit is a transformer-based multilingual toolkit for fundamental NLP tasks across 90 Universal Dependencies v2.5 treebanks and 56 languages. It improves performance while remaining fast on GPUs and efficient in memory use; future work targets additional pretrained transformers, named entity recognizers, and NLP modules.
- Conclusion: Trankit significantly improves fundamental NLP task performance across 90 Universal Dependencies v2.5 treebanks in 56 languages.The reported tasks include sentence segmentation, part-of-speech tagging, morphological tagging, and dependency parsing.
- Conclusion: Trankit is fast on GPUs and efficient in memory use, supporting use by general users.
- Future Work: Future work will investigate mBERT and XLM-Roberta-large and add named entity recognizers for more languages and modules for additional NLP tasks.