No-free-lunch theorem

A family of results showing that optimization algorithms perform equally when averaged over all possible objective functions. Any algorithmic advantage therefore depends on assumptions about the problems being solved.

Objective function: A function that assigns a score or cost to each candidate solution in an optimization problem. The theorem averages algorithm performance across possible objective functions.
Search-free lunch: An optimization result showing that a search strategy can outperform alternatives on a restricted, structured problem class. It demonstrates how assumptions about the problem class create genuine algorithmic advantages.
David H. Wolpert: A mathematician and computer scientist known for work on machine learning, optimization, and the no-free-lunch theorems. Wolpert formulated the influential results with William Macready.
Hyperparameter optimization: The process of selecting settings that control a machine-learning model’s training procedure. No-free-lunch reasoning cautions against expecting one tuning method to win on every model and dataset.
Occam's razor: The principle favoring simpler explanations when they account for the evidence equally well. Simplicity is a useful bias only when the task rewards the structures it favors.
Search algorithm: A procedure that explores a space of candidates to find solutions with desirable properties. The theorem compares algorithms by how they search candidate spaces.
Wolpert–Macready no-free-lunch theorems: A set of formal results comparing optimization algorithms across collections of objective functions. These theorems are the canonical formal statement of the no-free-lunch principle.
William G. Macready: A computer scientist whose work with David Wolpert established influential no-free-lunch results for optimization. Macready co-authored the foundational optimization theorems.
Metaheuristic: A general strategy for exploring difficult optimization problems without guarantees of universal superiority. Metaheuristics exploit particular structures or empirical regularities, not every possible objective.
Overfitting: A failure in which a model fits training data closely but performs poorly on new data. Without assumptions about data structure, observed optimization success need not transfer.

Linked from 7 pages