Overfitting

Overfitting occurs when a model captures details or noise specific to its training data and consequently performs poorly on new data.

Bias–variance tradeoff: The balance between errors from overly simple assumptions and errors from sensitivity to training data. Overfitting is the high-variance side of this tradeoff.
Machine learning: The study and practice of systems that learn patterns from data to make predictions or decisions. Overfitting is a failure mode across learned models.
Regularization: Techniques that constrain model fitting to discourage overly complex solutions. It directly penalizes patterns that fit training specifics without improving prediction.
Underfitting: A model's failure to capture patterns in data because its representation or training is insufficient. It is the opposite fit failure: training performance is poor rather than deceptively strong.
Model complexity: The capacity of a model to represent varied patterns in data. Greater capacity can fit training noise as well as genuine structure.
Empirical risk minimization: The procedure of choosing model parameters to minimize loss on observed training examples. Minimizing training loss alone can reward fitting noise.
Cross-validation: A procedure that repeatedly trains and evaluates models on different data partitions. Repeated held-out estimates help detect overfitting and select model settings.
Data shift: A change between the distributions of data used for training and those encountered later. It can impair new-data performance even when a model did not overfit.
Training set: The data examples used to fit a model's parameters. Overfitting is diagnosed when performance on these examples misleads about new data.
Loss function: A function that quantifies the error of a model's predictions. Training objectives and evaluation losses can expose memorization rather than useful prediction.

Linked from 24 pages