Most Internet Marketers are aware of how to implement A/B and Multivariate testing in Google Website Optimizer (and other tools) but lot of them doesn’t understand the importance of statistics in these experiments. Two terms that a landing page optimization (LPO) expert should understand before testing are: Statistical Significance and Confidence Level.
To understand statistics from an LPO point of view, let us take an example where we measure the conversion from the number of signup for our newsletter from 4 pages A, B, C and D, where A is the Control and B, C and D are the experiments:
From the above experiment, can we conclude that Landing Page D is the best performing page in the experiment? We can be certain only after evaluating two variables:
Confidence Level: is the probability that the results of the experiment can give similar results in the next one. So if we say that the confidence level is 90% then it means that there is 90% likelihood that if we repeat the experiment with similar parameters(landing pages and visitors), we will get the same winner (Landing Page D) with similar conversion rate(15%). A confidence level of 50% means that the results of the experiments are truly random, so we should aim for a confidence level of 95% to make sure that we can widely adopt the change in our website.
Statistical Significance: is the amount of data or in our case the number of conversions that would give us the confidence that a relationship does exist between two variables(Visitors and SignUp). For websites with unique monthly visitors above 50,000, you should look at 100 conversions. You need to understand that small difference in conversion (1-2%) might show significant difference when the sample size is much larger. Also, be careful in understanding what is statistically significant and significant enough from a Business point of view. Let us say that a conversion rate improvement of 2-5% would increase revenue by 20% but the experiment would take 2-3 weeks to gain statistical significance, it is in the Business interest to stop the experiment and implement change when you have found a better version of the landing page. You can always take the improved version of the landing page as the next control for your future experiments.
What is the best way to find the optimum combination of Confidence Level and Statistical Significance in Landing Page Optimization?
The answer is it depends on the unique visitors, the seasonality of your Business and the time frame when the experiment was conducted. For websites with millions of visitors, it is easy to test and reach statistical significance but for most websites, the visitors are way lower. The seasonality of the Business also impacts the experiment. For example, if you are conducting an experiment where conversion is measured as the number of Christmas gifts purchased then your entire experiments has to be measured 1 to 1.5 months before Christmas. The day and time of the week also impacts the experiment. For example, in some websites, Tuesdays and Wednesday tend to attract more qualified visitors than Saturdays and Sundays so experiments run on two different time frames would give different results but if we base our conclusion purely on confidence level then the results would still be wrong. So what is the best solution?
1) The best way to measure conversion rate is to plot the control and experiment results (conversion rate) in a Normal Distribution chart
2) Measure the standard deviation of the distribution
3) Understand the following properties of Normal Distribution:
68.26% of the results fall within 1 Standard Deviation around the mean.
95.46% of the results fall within 2 Standard Deviation around the mean.
99.74% of the results fall within 3 Standard Deviation around the mean.
Please note that Standard Deviation measures how much variation exists from the mean.
What we need to measure is how many standard deviations away from the mean is a specific conversion rate?
4) To answer the above question, you have to measure Z score
Z = (Conversion Rate - Mean Conversion Rate)/ Standard Deviation
Mean Conversion Rate = Sum of Conversion Rates/ Number of Samples
The Z Score will allow us to measure how confident we are about the results. Z Score can be positive and negative. We have to focus on Z(scores) that are +1 or above.
If the Z score is above 1.65 then we are 95% confident that the conversion rate of the experiment is 2 standard deviations away from the mean and the result can be implemented in the website.
If you want us to measure the results of your Landing Page Optimization Experiements and improve sales, contact us or you can call us at: India (Ph): +91 9497189032, UK (Ph): +44 (0)20-3371-9976 or US (Ph): +1 650-491-0004
ByteFive provides comprehensive Internet Marketing Services and Training in Search Engine Optimization, Search Engine Marketing (PPC Campaign Creation, Management and Optimization), Developing processes and best practices for Internet Marketing, Managing and Analyzing Web Analytics, Landing page Optimization, E-Mail Newsletter Management, Competitor Research, Content Creation and Content Marketing.