Natural language processing

Natural language processing (NLP) develops computational methods for analyzing, understanding, and generating human language. It combines linguistic knowledge with statistical and machine-learning techniques.

Tokenization: The process of splitting text into units such as words, subwords, or characters. It defines the units that many NLP models process.
Machine translation: The automatic conversion of text or speech from one language into another. Translation is a longstanding NLP task with both statistical and neural approaches.
Noam Chomsky: An American linguist whose theories of grammar reshaped linguistics and influenced early language technology. His account of formal grammar helped frame early questions about computational language.
Rule-based machine translation: A translation approach that relies on hand-written linguistic rules and dictionaries. It contrasts with systems that infer translation patterns from examples.
Hallucination (artificial intelligence): The production of plausible but unsupported or false content by a generative AI system. Fluent generation does not guarantee that an NLP system's claims are grounded.
Word embedding: A numerical representation of a word that encodes aspects of its meaning and usage. Embeddings let models use semantic and syntactic patterns in text.
Sentiment analysis: The computational identification of opinions, attitudes, or emotions expressed in text. It applies language analysis to judgments expressed in reviews and posts.
ELIZA: A 1960s conversational program that used pattern matching to simulate dialogue. ELIZA became an early demonstration of computer-generated conversation.
Symbolic artificial intelligence: An approach to AI that represents knowledge with explicit symbols and rules. Its explicit structures differ from statistical NLP's learned parameters.
Bias in artificial intelligence: Systematic distortions in AI outputs that can produce unfair or unequal effects across groups. Training data and model design can reproduce or amplify social biases in language.

Linked from 29 pages