split testingtesting changes iteratively

Split testing is the process of showing different versions of a page to different visitors.  The performance of the different versions are measured and, if one out performs the other, it then becomes the new version of the site.  Companies like Amazon and Google split test continuously and are always looking for improvements.

Split testing is also known as A/B testing and is closely associated with multivariate testing. It is an extremely important process in the development of any site as any suggested design or content changes up to this point are just theories or personal opinions. How do you know a change will actually work and improve anything?

Although it is rare that theories based upon empirical evidence and user testing are wrong, it is common that the theory is only applicable to a small proportion of a user base and only results in a small overall change to conversion rate. In the worst case implementing a theory can lead to a decrease in conversion rate – sometimes the case with sweeping changes as there are so many variables being affected.

An ongoing testing process

Testing in a step-wise manner (as opposed to making a host of changes at the same time) ensures that only the beneficial changes are implemented and will also inform the ongoing process of conversion rate optimisation. Anything that reduces the number of changes being made on gut instinct or ‘flavour of the day’ decision alone is a good idea and can avoid expensive ‘undoing’ of disastrous changes.

The process in brief

• Hypothesis preparation – Actionable hypotheses based upon experience and insights from user testing or advanced analytics
• Test variation creation – Creation of improved variations of your current pages.
• Test setup – Deliver variations of the pages to your different users at the same time to see which variation of the page works better
• Test evaluation – Reporting on the outcomes of testing as soon as sufficient statistically-significant data has been gathered