Linked from

The 10 pages that link to Tokenization, each with the reason it gives.

• Natural language processingRelated: It defines the units that many NLP models process.

• Search engineRelated: Search engines tokenize documents and queries before matching their terms.

• Machine translationRelated: Translation models operate on tokens, and token boundaries affect vocabulary and output quality.

• Mobile paymentRelated: Wallets use payment tokens so merchants need not receive the card number stored on a device.

• Large language modelRelated: A model predicts token IDs rather than directly predicting whole words or sentences.

• Sentiment analysisRelated: Models need a representation of text before they can classify it.

• Word (linguistics)Related: Computational word counts require explicit decisions about what counts as a boundary.

• Computational linguisticsRelated: It turns raw text into units that many downstream language algorithms can process.

• ChatGPTRelated: ChatGPT processes prompts and generates replies as sequences of tokens.

• Heaps's lawRelated: Different token boundaries change both the total word count and vocabulary size.