A Guide to A/B Testing

What is A/B Testing?

A/B testing is one of the most essential ideas in data science and the IT sector in general since it is one of the most effective strategies for determining the validity of any hypothesis.

In its most basic form, A/B testing is a comparison of two versions to evaluate which performs better based on a certain measure. Typically, two consumer groups are exposed to two alternative versions of the same product to determine if metrics such as sessions, click-through rate, and/or conversions change significantly.

A/B testing, to be more precise, is a type of statistical and two-sample hypothesis testing. Statistical hypothesis testing is a method for comparing a sample dataset to population data. On the other hand, the process of assessing if the differences between the two samples are statistically significant is known as two-sample hypothesis testing. In this specific situation, the population refers to the total number of consumers who have purchased our product or visited our website, whereas the _sample refers to the number of customers who have taken part in the test.

In the figure above, we can see that we have divided our sample space into two separate groups, i.e. control group and variant group. We then show the variant group the new design and observe whether conversions rise significantly. When doing an A/B test, keep in mind that all other variables must be kept constant.