Byte-Pair Encoding (BPE)

Byte-Pair Encoding (BPE) is a subword tokenization algorithm at the heart of most modern large language models. The core idea is simple: start with a base vocabulary of individual characters or bytes, then repeatedly find the most frequent adjacent pair in the training corpus and merge it into a new vocabulary entry. The procedure stops when a target vocabulary size — typically 30k to 200k entries — is reached. BPE began as a generic data compression algorithm published by Philip Gage in 1994. Rico Sennrich, Barry Haddow, and Alexandra Birch repurposed it for neural machine translation in their 2016 ACL paper 'Neural Machine Translation of Rare Words with Subword Units', solving the open-vocabulary problem by letting models compose rare words from learned subword pieces. OpenAI's tiktoken library implements a byte-level variant, where merges operate on UTF-8 bytes rather than Unicode code points, guaranteeing that any input string can be encoded losslessly. BPE's strengths are speed, simplicity, and reasonable compression ratios across languages. Its weaknesses include sensitivity to whitespace and casing, uneven token efficiency across languages (non-Latin scripts often need many more tokens per word), and the fact that learned subword boundaries can hide character-level structure from the model — the root of the Tokenization Boundary Problems in LLMs. Variants such as byte-level BPE in GPT-2/4 and BPE-dropout for regularization are now standard.