A/B Testing Explained

Consider this situation: You run the marketing department for an e-commerce outdoor outfitter, and your team has decided on a subject line and special offer for your next campaign. The variable you're trying to settle is whether the main graphic should be a product photo or one of the company president announcing the offer. This is the perfect time to employ an A/B split to find out which works better.

Creating campaign A
First create the e-mail using product shots — in this case, one that includes an array of men's and women's shoes. Then, create a landing page that reiterates the president's sale coupon offer, in addition to using the same product imagery.

This will be your ‘A' campaign.

Creating campaign B
In order to isolate how much impact the imagery has on the recipient's actions, campaign ‘B' will be an exact replica of ‘A,' but in place of the shoes photo, use a photo of your president. The landing page includes the coupon offer and that same picture.

Randomize, send and analyze
Now it's time to send out the e-mail. If your e-mail service provider offers you the option to randomly select half of your list, use it. If not, simply cut your list in half alphabetically or by order of the opt-in date, bearing in mind that the results will be slightly skewed because of these variables.

After you've sent the mail, look at your metrics. First, check your e-mail stats to determine if your click-through rates are better for either campaign. Then, use your Web analytics application to see how visitors from campaign A behaved differently from visitors from campaign B. And although you'll want to concentrate on conversion rates, you can glean knowledge from other metrics, as well. How did the visitors' average time on site differ? Did visitors from one campaign view a vastly different number of pages than visitors from the other? Contrasting and comparing how each campaign's visitors behaved lets you draw conclusions about what worked and what didn't.

Drawing conclusions
In our imaginary scenario, let's assume that you found that campaign B (featuring the picture of the president) yielded 1.5 times better purchase conversion than campaign A, which featured product shots. Your customers showed you that seeing your company president made them feel more compelled to purchase.

A/B testing may be a bit more work on the front end of an e-mail campaign, but the information and insights you gain from it can make a huge difference in your click-through rates, as well as your conversion rates. So the next time your team can't decide between two good ideas, split the difference and let your customers show you.

Source: DMNews

Comments

[Tim Wilson]

Two other points to add on:

Depending on the size of your list and the time you have to execute the campaign, ideally, you would do the A/B test with a subset of the list, determine which performed better, and then execute the full campaign with the better-performing message/copy. This requires that you have a large enough overall list so that you still have enough people left over after the A/B test to make it worthwhile.

And...defining "enough" goes to the overall power and significance of the results. It seems awfully common for marketers to simply look for the higher percentage in an A/B test and jump to a conclusion that one message performed better than others. If you understand that part of your results are due to "noise" (they always are, even in the most controlled of experiments), then you know the size of the test and how much of a difference you see need to be factored into assessing the results. JT Buser at Bulldog Solutions put together a couple of free, downloadable Excel spreadsheets to help with that assessment: http://www.bulldogsolutions.com/ExcelABSplitcalculator/.

[Tamara Gielen]

Hi Soeren,

Thanks for you comment, I totally agree with you - splitting a list based on opt-in date is definitely going to skew your data. The best way to split a list is to do a random split. If your email service provider doesn't offer the option to do a random split, I'm pretty sure Excel can do it for you.

Tamara

[Soeren Sprogoe]

Hi Tamara,

first time reader, first time poster here :-)

Excellent article! However there's one point where I strongly disagree:

If you order your list by subscription date, and then cut it in half, you'll get:

- List A with either your best customers, or a ton of deadweight from deadend email addresses that hasn't bounced for some reason.

- List B with all the newest email addresses. These might be extremely ready to convert as your company/product is still fresh in their minds, or they might not yet be ready to convert.

Which is the case on list A/B very much depends on your industry and recruitment strategy. But my point is that splitting by subscription date will give you a completely wrong result. Try sending out exactly the same email to these two lists and see what happens.

To me, the most important thing to do a valid A/B-test is how you do the split. But then again, I work as a Web Analyst, so I gotta say that my job is the most important one :-)