Experimentation Calculator
Compare control and variant conversion rates with a two-proportion z-test, confidence interval and an estimated sample-size check before declaring a winner.
Use mutually exclusive control and variant groups measured over the same experiment window.
A statistically significant result means the observed difference is unlikely under the null hypothesis at the confidence threshold you selected. It does not prove the variant will keep the same lift forever, and it does not tell you whether the lift is commercially meaningful.
What statistical significance does — and does not — mean
The calculator uses a two-sided two-proportion z-test. The p-value describes how surprising the observed difference would be under a no-difference assumption. It does not tell you the probability that a variant will keep winning in the future.
The confidence interval shows a plausible range for the absolute conversion-rate difference under the model. A narrow interval around a tiny lift can be statistically convincing but commercially unimportant.
Standard fixed-horizon tests assume the decision rule was defined before the data was repeatedly inspected. Continuous peeking, stopping on a favourable day or changing the primary metric mid-test increases the chance of a false positive.