SentencePiece

SentencePiece is a language-independent subword tokenization toolkit released by Google researchers Taku Kudo and John Richardson at EMNLP 2018. Unlike earlier tokenizers that required language-specific word splitting before subword training, SentencePiece trains directly on raw byte or Unicode sequences and treats whitespace as just another character (commonly encoded as the meta-symbol '▁'). This makes the same training pipeline work for English, Chinese, Japanese, and code without special pre-processing. The library supports two main algorithms: Byte-Pair Encoding (BPE)|BPE, which greedily merges frequent pairs, and the Unigram language model, which starts from a large candidate vocabulary and prunes it to maximize corpus likelihood. The Unigram option also enables subword regularization, where the tokenizer samples among plausible segmentations during training, improving robustness. SentencePiece is open-sourced under the Apache 2.0 license and ships C++ and Python bindings. It has become a de facto standard outside the GPT family: Meta's LLaMA series, Google's T5 and mT5, and Gemini all use SentencePiece-trained vocabularies, while OpenAI maintains its own tiktoken BPE stack. Despite differences in vocabulary and algorithm, all share the basic property that the model sees subword IDs rather than characters — see Tokenization Boundary Problems in LLMs for the downstream consequences.