Multiple comparisons problem

The increased chance of obtaining false-positive results when many statistical hypotheses are tested, especially when each test is judged at an unadjusted significance threshold.

Family-wise error rate: The probability of making at least one false rejection among a family of statistical hypotheses. It measures the accumulated false-positive risk that defines the problem.
Hypothesis testing: A statistical framework for evaluating claims about populations using sample data. Each comparison adds another test whose error contributes to the problem.
Genome-wide association study: An observational study that tests genetic variants across the genome for association with a trait. Testing many variants demands stringent control of false-positive findings.
Preregistration: The public recording of a study's questions, methods, and analysis plan before outcomes are examined. It limits undisclosed analysis choices but does not itself correct multiplicity.
P-hacking: The practice of trying multiple analyses or data choices and selectively reporting those producing statistically significant results. Unreported analytic flexibility can turn many implicit comparisons into false discoveries.
Bonferroni correction: A method that controls family-wise error by testing each hypothesis at a reduced significance threshold. It limits the total false-positive probability by dividing the threshold across tests.
Null hypothesis: A statistical claim treated as the default for testing, often specifying no effect or difference. False positives occur when a true null hypothesis is rejected.
Microarray analysis: A method for measuring the expression of thousands of genes simultaneously. Thousands of gene-level tests create a large multiple-testing burden.
Independent replication: A new study that tests a finding using independently collected data and methods. Replication checks whether a selected finding survives a fresh test rather than a correction.
Publication bias: The systematic tendency for published research to favor particular results, often statistically significant findings. Selective publication can amplify the apparent success of chance findings.

Linked from 33 pages