Between projects we like to set up A/B testing experiments on our website which test theories and ideas that come up during client work. Our latest experiment focused on how to encourage users to download a free guide by testing the differences between a large call to action box versus a smaller one. Rather than worry too much about getting the design and styling right, first we wanted to test out which would generate the most conversions and then adopt an ongoing agile approach to eventually lead us to the right solution.
In true ‘which test won’ style, take a look at these two pages and take a stab at which test you think would convert better (the winner will be revealed later in the article):
Before we went live with our test, we spent some time plotting out the user journey and the key steps visitors would go through to get the customer profiling guide (the conversion goal). We brainstormed different mechanisms to capture email addresses which would not take users away from the page they were on. We felt that a button which triggered an overlay would allow users to complete the process in context without having to move away from the content they had originally visited.
During the wireframing stage we felt that the larger call to action box (Test A) was more likely to convert because it draws user attention and offers more detail on the benefits of downloading the free guide. We wanted to test this alongside a smaller, less intrusive box (Test B) which did not push the page content down and take up too much space.
With our hypothesis set, we quickly converted our wireframe mockups from the whiteboard into html. Within a couple of hours our two pages were set up, the tracking code was implemented and google experiments was up and running; sending our traffic to the two versions of the what is customer profiling? page.
After a couple of weeks of letting the tests run we were surprised to find that the smaller box (Test B), outperformed the large box with almost double the amount of conversions:
This was interesting because typically in usability testing we see that larger, clearer choices for users perform better than the more subtle ones. This just goes to show how dangerous assumptions can be in web design.
A/B testing is excellent at telling you the facts, but it has no ability to answer that all important question: why did the smaller box convert better than the bigger box?
So, whilst we didn’t fully understand why the smaller call to action box worked better, we felt that we had enough evidence to conclude that the next set of tests should be to test the wording used in the box, and then after that test was complete, we could look at the design and styling of the box and button.
We’re now testing a simple wording change to see if the change in emphasis from ‘our’ to ‘your’ makes a difference. We’ll update you with the results once we’ve completed the experiments.
What we find really valuable in our agile A/B testing approach is that we can quickly test out our ideas without having to worry about perfecting each option. We focus on the overall direction and then with each experiment we can refine the details until we have a direction we are confident in. Once we have that, we can roll it out to the rest of our site in A/B tests against our existing content versus the winning, fine-tuned design from these tests.
We wanted to share this approach as we often find that our clients want to polish everything and make their options perfect before they are tested. Yet we find there is much more value in testing rapidly and advancing incrementally rather than spending too much time perfecting two different options.
Are you spending too much effort perfecting rather than testing? Then give us a call. We are experts in conversion rate optimisation and usability testing and can help you find the best solution for your business.
Damian is our Innovation Director who has 16 years experience in UX working for companies like the BBC and National Air Traffic Services researching & designing websites, apps, voice recognition, and air traffic control interfaces. With two kids under two he spends his rare free time catching up on lost sleep or immersed in his xbox.