P-value

A p-value is the probability, under a specified null hypothesis and statistical model, of observing results at least as extreme as those obtained. It measures compatibility with that model, not the probability that the null hypothesis is true.

Null hypothesis: A statistical claim treated as the reference condition for evaluating observed data. The p-value is calculated assuming this hypothesis and its model are true.
Sampling distribution: The probability distribution of a statistic across repeated samples drawn under specified conditions. A p-value locates the observed statistic within the distribution expected under the null.
Bayes factor: The ratio of two models’ likelihoods for observed data, used to compare their relative support. Unlike a p-value, it compares specified hypotheses rather than measuring tail probability under one null.
Karl Pearson: An English mathematician and statistician who developed foundational methods in mathematical statistics. His chi-squared test helped establish probability-based assessment of discrepancies from a model.
P-hacking: The use of analytical choices or repeated testing to obtain statistically significant results. Selective analysis can make reported p-values appear smaller than a prespecified analysis would yield.
Test statistic: A numerical summary of data used to evaluate a statistical hypothesis. The observed statistic is compared with its null distribution to calculate the p-value.
Statistical significance: A designation for results whose p-values fall below a chosen threshold. The threshold turns a continuous p-value into a conventional decision label.
Posterior probability: The probability of a hypothesis after combining prior probabilities with observed data. A p-value is not the probability that the null hypothesis is true.
Ronald Fisher: A British statistician who developed influential methods for estimation, experimental design, and significance testing. He popularized the use of small p-values as evidence against a null hypothesis.
Publication bias: The selective publication of research findings based on their direction, magnitude, or statistical significance. Preference for significant results distorts the evidence represented in published p-values.

Linked from 32 pages