A/B testing

A controlled experiment that randomly assigns people to two versions of a product, design, or message and compares measured outcomes.

Randomized controlled trial: An experiment that randomly assigns units to treatments to estimate their effects. A/B testing is a randomized controlled trial adapted to product and communication decisions.
Hypothesis testing: A statistical framework for evaluating data against a prespecified claim about a population or process. A/B results are often assessed by testing whether version outcomes differ beyond chance variation.
Website optimization: The practice of improving a website's performance, usability, or business outcomes through design and technical changes. A/B tests compare page layouts, wording, and interaction choices against defined site goals.
Multivariate testing: An experiment that tests combinations of multiple factors or component variations at once. It estimates interactions among elements, while a basic A/B test compares two complete versions.
Random assignment: A procedure that allocates experimental units to groups by chance. Chance-based allocation helps make the two groups comparable before exposure to either version.
Null hypothesis: A statistical claim, often that an effect or difference is absent, used as a reference in hypothesis testing. The usual test asks how surprising the observed difference would be if the versions performed equally.
Email marketing: The use of email messages to communicate with audiences and pursue marketing objectives. Subject lines and message content can be tested by randomly splitting mailing lists.
Quasi-experiment: A study estimating intervention effects without random assignment of units to conditions. It is an alternative when product changes cannot be assigned randomly.
Control group: A group that does not receive the treatment being evaluated or receives a baseline condition. The existing version commonly serves as the control against the proposed alternative.
Statistical power: The probability that a statistical test detects an effect of a specified size when that effect exists. Power planning helps avoid interpreting an inconclusive small experiment as proof of no effect.

Linked from 21 pages