And the results can be measured and clearly communicated to customers. There are hundreds of success stories with A/B testing, such as an eCommerce strategy that generated 36% more cart closures or a marketing test in a healthcare service that increased leads by 638%.
But that level of results is not easy to achieve. In working with companies to optimize conversion using A/B testing we have observed some commonalities in the challenges faced by companies, whether large or small, B2B or B2C, e-commerce or lead generation.
Challenge 1: Knowing what to test
You can't put out two versions of the same communications piece and simply expect a lift in engagement metrics. Marketers quickly learn that some small changes that are attractive because they are easy to implement are not impactful enough to drive change. They also know that dramatic modifications can result in losses.
To know what to test, you must first know where to test. This is where your customer data can be very decisive. Run an analysis of metrics within the lifecycle, sales funnel or type of communications you do, to move them from one state to another and identify where they are filtered. Where do potential customers fall?
Challenge 2: Running valid tests
Here one could make the mistake of believing that one has discovered a formula to increase conversion, when in fact this is not the case. Another possible mistake is to overlook a conversion increase.
A/B testing is a successful tactic because of its predictive power. For the test results to play a truly predictive role, you must ensure that they reflect the customer's behavior and that it was the change made in the test that triggered the new results.
To achieve this, the experiment must be set up and monitored in a scientific manner and avoid threats to validity such as:
- Deliverability effects: for example, 10,000 messages were not delivered due to a server malfunction.
- Effects of history: for example, unexpected advertising around the product at the exact time the test is run, a marketing campaign that skews demand temporarily in one direction, running a test for only 20 hours on a Tuesday when weekend traffic behaves very differently, or running a test on your e-commerce site with highly motivated December vacation traffic and expecting to get the same results in January.
- Selection effects: for example, another division runs a pay-per-click ad that drives traffic to a landing page in their campaign at the same time you run your test, or customers automatically select which treatment they see.
- Sampling of distortion effects: This is a failure to collect a sample size sufficient to overcome the random possibility. For example, determining that a test is valid based on 100 responses.
Challenge 3: Interpreting Test Results
Let's assume that you successfully accomplish points 1 and 2 and get a great result. There is still a fundamental question to be answered: Why? Why do customers behave the way they do? What have you learned about the customer and how can you use this knowledge?
The interpretation of the results of a test should not be limited to the moment immediately after its application. In fact, the key to interpreting the results lies just prior to their execution. It is about establishing the hypothesis with the goal of thoroughly understanding the customer's thought processes at crucial delivery points in your funnel, so that you can increase perceived value and decrease perceived cost.
By understanding what prospects are thinking at each stage of the buying process, you'll be in a better position to match their motivation and move them through the sales funnel faster.
The goal should not be to simply increase a KPI (key performance indicator), but to understand how you can better serve the customer with your marketing messages, sales process and even products. And through that understanding, you will improve results.
So even if your test treatment ends up producing fewer conversions, it's not really a loss, because you've gained customer insight. If data is the currency in the Internet age, customer insight is the gold standard: the central piece of data to which all marketing must be directly linked.
And, ultimately, that's how you get long-term increases with A/B testing.
