11. Project: A/B Test Report

An online store spent two weeks testing a new checkout page. Management doesn't want "p is below 0.05". They want a clear answer: how much the metrics changed, how sure we are, and what to do. For that, instead of a single p-value you calculate confidence intervals (CIs): the range in which the true effect lies with the chosen confidence (95% at α = 5%).

• the critical z for a two-tailed interval is NORM.S.INV(1-α/2), about 1.96 at α = 5%;
• the CI of the difference in conversion rates: (pB - pA) ± z·SE, where SE = √(pA(1-pA)/nA + pB(1-pB)/nB). Here the variances are calculated for each group separately, so the interval describes the actual difference rather than the "no difference" hypothesis;
• the margin of error of the average order value: CONFIDENCE.NORM(alpha, standard_dev, size) = z·s/√n. For small samples there's CONFIDENCE.T, which is slightly wider;
• the CI of the difference in average order values: (valueB - valueA) ± z·√(sA²/nA + sB²/nB);
• revenue per visitor = conversion rate x average order value. It combines both metrics. A new page can raise conversion but drag down order value.

How to read an interval: entirely above zero means B is definitely better; entirely below means worse; containing zero means there isn't enough data to conclude. The width of the interval shows how precise the estimate is.

The decision rule is set in advance: "ship B" if the CI of the conversion difference is above zero and B's revenue per visitor is higher; "keep A" if the CI of the conversion difference is entirely below zero; otherwise "need more data".