Tokenization

Tokenization divides text into units called tokens for computational processing. Depending on the method, tokens may be words, subwords, characters, or other text segments.

Whitespace tokenization: A tokenization method that splits text at spaces and other whitespace characters. It is the simplest boundary rule, but punctuation and languages without spaces expose its limits.
Vocabulary (language models): The set of token types that a language model can represent directly. A tokenizer's rules and vocabulary jointly determine which text units receive dedicated token IDs.
Large language model: A neural network trained on large text datasets to process or generate language. Its input and output are token sequences defined by a tokenizer.
Character-level language model: A language model that represents text as individual characters rather than words or subwords. It avoids a fixed word vocabulary but generally requires longer sequences than subword tokenization.
Byte-pair encoding: A subword tokenization method that iteratively merges frequent adjacent symbol pairs. Its learned merges let language models represent common words compactly and split rare words.
Token ID: An integer assigned to a token in a model's vocabulary. Tokenization turns text into token IDs before a model can process it.
Context window: The maximum sequence of tokens a model can consider in one processing pass. Different tokenizers fit different amounts of the same text within this limit.
Byte-level encoding: A text representation that encodes data as bytes rather than linguistic units. Byte-level methods guarantee coverage of arbitrary text but can use more sequence positions.
WordPiece: A subword tokenization algorithm that builds a vocabulary from pieces useful for representing text efficiently. BERT uses WordPiece to balance vocabulary size with coverage of unfamiliar words.
Embedding (machine learning): A learned numerical representation of an item, such as a token, in a vector space. Models map token IDs to embeddings before processing their sequence.

Linked from 10 pages