Scaling law

A scaling law is an empirical relationship between a model’s performance and resources such as training compute, data, or parameter count. It helps predict how performance changes as those resources increase.

Neural scaling laws: Empirical power-law relationships describing how neural-network loss changes with model size, data, and compute. This is the central family of scaling relationships used to forecast language-model training.
Scaling Laws for Neural Language Models: A 2020 study by Jared Kaplan and colleagues measuring language-model loss against parameters, data, and compute. It established influential power-law fits and a compute-allocation rule for language models.
Data quality: The degree to which a dataset is accurate, relevant, diverse, and suitable for a task. Equal token counts can produce different outcomes when the training data differs in quality.
Compute governance: Policies and institutions governing access to, use of, and oversight over advanced computing resources. Forecasts of capability growth with compute inform proposals for tracking and controlling large training runs.
Power law: A relationship in which one quantity varies as a constant times a power of another. Many observed scaling curves are approximated by power laws over practical ranges.
Training Compute-Optimal Large Language Models: A 2022 DeepMind study showing that compute-optimal language models should use more training tokens than prior practice suggested. Its Chinchilla results revised common assumptions about the best model-size and data allocation.
Data contamination: The presence of evaluation examples or their close equivalents in a model’s training data. Contamination can make measured performance look better without reflecting genuine generalization.
AI capability evaluation: The systematic measurement of what artificial-intelligence systems can do across tasks and conditions. Scaling forecasts require evaluations that detect performance changes reliably.
Cross-entropy: A loss function measuring the difference between predicted probabilities and target outcomes. Language-model scaling studies often measure performance as held-out cross-entropy loss.
Chinchilla: A 70-billion-parameter language model trained on about 1.4 trillion tokens by DeepMind in 2022. It demonstrated the performance gains from training a smaller model on substantially more data.

Linked from 8 pages