Cross-validation

Cross-validation estimates how well a predictive model generalizes by repeatedly fitting it on one portion of data and evaluating it on another.

K-fold cross-validation: A procedure that divides data into k folds and uses each fold once for evaluation while training on the others. Its rotating holdout folds are the standard implementation of cross-validation.
Training set: The data used to fit a predictive model. Each cross-validation iteration fits the model using its training portion.
Hyperparameter optimization: The process of selecting model settings that are not learned directly from training examples. Cross-validation scores are commonly used to compare candidate hyperparameter settings.
Holdout method: An evaluation procedure that trains on one data partition and tests on a separate partition once. Unlike cross-validation, it does not rotate the evaluation partition across repeated fits.
Stratified sampling: A sampling method that preserves chosen subgroup proportions across samples. Stratifying folds can keep class proportions similar when outcomes are imbalanced.
Test set: Data reserved to evaluate a fitted model on examples not used for fitting. Each held-out fold serves as a temporary test set.
Model selection: The process of choosing a predictive model from a set of candidates. Applying the same folds to candidates makes their estimated performance easier to compare.
Bootstrap aggregating: An ensemble method that averages predictions from models fitted on bootstrap-resampled data. It resamples training data to reduce prediction variance rather than primarily estimating performance.
Leave-one-out cross-validation: A cross-validation method that holds out one observation at a time for evaluation. It is the limiting case of k-fold cross-validation with one fold per observation.
Generalization error: The expected prediction error of a model on new data drawn from its target population. Cross-validation approximates this error using held-out observations.

Linked from 15 pages