Source-linked AI summary
SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing
Taku Kudo, John Richardson
TL;DR
Language-dependent preprocessing limits end-to-end, language-independent neural text processing. SentencePiece trains and applies subword models directly to raw text, achieving 380× faster segmentation than subword-nmt on raw Japanese data.
Problem
Many neural machine translation systems still rely on language-dependent preprocessing, motivating simple, reproducible, language-independent text processing.
Method
SentencePiece performs lossless subword tokenization directly on raw Unicode text, preserves whitespace information, and converts text to vocabulary-indexed sequences.
Results
SentencePiece segmentation is about 380 times faster than subword-nmt on raw Japanese data without pre-tokenization.
Takeaways & Limitations
SentencePiece supports purely data-driven, language-independent neural text processing without requiring pre-tokenization.
Takeaways & Limitations
The representation cannot encode consecutive whitespaces, so lossless tokenization is not always possible.
Abstract
from arXiv · showhide
This paper describes SentencePiece, a language-independent subword tokenizer and detokenizer designed for Neural-based text processing, including Neural Machine Translation. It provides open-source C++ and Python implementations for subword units. While existing subword segmentation tools assume that the input is pre-tokenized into word sequences, SentencePiece can train subword models directly from raw sentences, which allows us to make a purely end-to-end and language independent system. We perform a validation experiment of NMT on English-Japanese machine translation, and find that it is possible to achieve comparable accuracy to direct subword training from raw sentences. We also compare the performance of subword training and segmentation with various configurations. SentencePiece is available under the Apache 2 license at https://github.com/google/sentencepiece.
1 Introduction
The introduction identifies a gap between end-to-end NMT and its continued reliance on language-dependent preprocessing, especially for non-whitespace-segmented languages. It presents SentencePiece as a simple, efficient, reproducible, language-independent tokenizer and detokenizer that supports subword modeling directly from raw sentences.
- Motivation: Many NMT systems still rely on language-dependent preprocessing and postprocessing inherited from traditional SMT systems.These tools use hand-crafted rules whose effectiveness for NMT has not been proven.
- Motivation: Existing preprocessing tools are mainly designed for European languages with whitespace-delimited words, creating difficulties for Chinese, Korean, and Japanese.Moses is described as a de-facto standard SMT toolkit, but it is built on language-dependent rules.
- Motivation: The field therefore needs a simple, efficient, reproducible, and language-independent preprocessor and postprocessor for neural-network-based NLP systems, including NMT.This need grows as NMT architectures become more language-agnostic and standardized.
- SentencePiece: SentencePiece is introduced as a simple, language-independent tokenizer and detokenizer for neural-network-based text generation systems with a predetermined vocabulary size.It is designed mainly for systems where vocabulary size is fixed before neural model training.
- SentencePiece: SentencePiece implements BPE and unigram language-model subword segmentation with direct training from raw sentences.This extension supports subword modeling without requiring pre-tokenized input.
2 System Overview
SentencePiece consists of four components that normalize text, train and apply subword models, and decode the resulting representations. Its encoder and decoder directly map text to and from vocabulary id sequences, enabling reversible end-to-end processing.
- Components: SentencePiece comprises four components: Normalizer, Trainer, Encoder, and Decoder.The Normalizer canonicalizes semantically equivalent Unicode characters, while the Trainer learns subword segmentation from the normalized corpus.
- Encoding and Decoding: The Encoder and Decoder perform tokenization and detokenization while managing vocabulary-to-id mappings.They can directly convert text into id sequences and vice versa, which is useful for NMT systems with id-sequence inputs and outputs.
- End-to-End Workflow: SentencePiece reversibly converts input text through spm_encode and spm_decode.The end-to-end workflow includes training with spm_train, encoding with spm_encode, and decoding with spm_decode.
3 Library Design
SentencePiece uses lossless, language-independent tokenization by preserving whitespace and treating input as Unicode characters, while supporting configurable vocabularies, normalization, reproducibility, and library integration. Its design addresses irreversible language-dependent preprocessing, scalability, and limitations of pre-tokenized workflows.
- Lossless tokenization: Language-dependent tokenization loses spacing information and requires costly hand-crafted detokenization rules, especially across languages with different spacing conventions.The tokenized sequence cannot preserve whether a space existed before punctuation, while Japanese generally requires no spaces between primitive tokens.
- Lossless tokenization: SentencePiece preserves whitespace as a Unicode symbol, enabling lossless tokenization and unambiguous detokenization from arbitrary subword sequences.Whitespace is escaped with the meta symbol ▁ (U+2581), and the decoder reverses the encoder operation.
- Raw-text training: SentencePiece trains directly from raw sentences, avoiding unavailable pre-tokenization for non-segmented languages and improving compatibility with lossless tokenization.Existing tools train from pre-tokenized sentences, but pre-tokenization is not always available and makes lossless tokenization difficult.
- Efficiency: BPE segmentation is reduced from O(N^2) naive cost to O(N log(N)) by managing merged symbols with a binary heap.The optimization supports training and segmentation on large amounts of raw data.
- Vocabulary and metadata: SentencePiece specifies the final vocabulary size and manages vocabulary-to-id mappings, while reserving configurable ids for special and custom meta symbols.This vocabulary-size design is applicable beyond BPE to algorithms such as the unigram language model.
- Normalization and reproducibility: The model file bundles vocabulary, segmentation parameters, and a pre-compiled normalization transducer, making behavior self-contained and reproducible without external dependencies.By default, input is normalized with Unicode NFKC, although SentencePiece implements only a subset because full CCC reordering is difficult in a finite state transducer.
4 Experiments
Experiments on English–Japanese KFTT translation compared word-based and SentencePiece preprocessing under GNMT, showing consistent BLEU gains from subword segmentation and especially strong benefits on raw Japanese data. SentencePiece also achieved major segmentation-speed improvements over subword-nmt for raw Japanese input.
- Experimental setup: The experiments evaluated English–Japanese Wikipedia translation on KFTT using GNMT with 512-unit, 6-layer LSTMs.KFTT contains 440k training, 1166 development, and 1160 test sentences.
- Experimental setup: SentencePiece unigram models were compared with and without pre-tokenization against a word-model baseline using case-sensitive BLEU.Moses and KyTea were used for English and Japanese pre-tokenization, respectively; Japanese outputs were segmented with KyTea before scoring.
- Translation results: SentencePiece subword segmentation consistently improved BLEU over the word model, while pre-tokenization was unnecessary in some directions and degraded English-to-Japanese performance.For Japanese-to-English translation, the improvement from pre-tokenization was marginal and not statistically significant.
- Translation results: Larger BLEU improvements occurred when SentencePiece was applied to Japanese and when Japanese was the target language.The paper attributes this to raw-input unsupervised segmentation finding domain-specific Japanese vocabulary without the strong constraint imposed by pre-tokenization.
- Performance comparison: SentencePiece segmentation was about 380 times faster than subword-nmt on raw Japanese data without pre-tokenization.On English data, SentencePiece and subword-nmt had almost comparable training and segmentation speeds regardless of pre-tokenization.
5 Conclusions
The paper introduces SentencePiece as an open-source subword tokenizer and detokenizer for neural text processing, enabling end-to-end text-to-id conversion without language-specific resources. Its self-contained model files support perfectly reproducible normalization and subword segmentation.
- SentencePiece is an open-source subword tokenizer and detokenizer designed for neural-based text processing.
- Direct text-to-id conversion helps build purely end-to-end systems without relying on language-specific resources.
- Self-contained model files guarantee perfect reproducibility of normalization and subword segmentation.