A guide to creating A/B tests for your organization’s emails

Email has become, and continues to be, a powerful medium for reaching your supporters. As such, you need your emails to be the most effective that they can be. So how do you do that? By creating different versions and sending test samples of your emails to small groups of your supporters.

So what is A/B Testing?

A/B Testing is the practice of testing small changes in emails to see which version is more effective. Another name you may have heard for A/B Testing is split testing. You can run these tests on more than just emails too – you can do it for web pages, donation forms, sign up forms, or any other way you interact with your supporters online.

While I will just be talking about A/B Testing for emails, you can apply much of this to other types of A/B Testing as well.

The important thing to remember about an A/B Test is that you don’t want to change a bunch of things because there will be no way to determine which of your changes lead to a better outcome. Instead, you should focus on making one change to find out what is most effective for your audience. While A/B Testing may take a little more time and effort on your part, the insight it provides is well worth it.

What kind of changes can you test?

There is a huge list of things you can change to create an A/B Test. Some ideas to get you started:

• —Change subject lines
• Try different pictures
• —Make one longer messaging version, one shorter messaging version
• Change where you normally place your action button
• —Change the size of your button
• —Change the color of your button
• —Change who the message is from
• —Change time/date the message is sent

This is just a short list I created, but I’m sure you can think of a seemingly endless list of things you could change in your own emails. The point is, find what you think could make the biggest difference to your supporters and test to see if you’re right. This way, you are finding what your supporters have the best response to and making more effective messages to communicate with them.

How do you set up an A/B Test?

Now that you have an idea of what you can do with A/B Testing, let’s get into how you actually go about running one. It breaks down 5 simple steps:

—Step 1: Make 2 identical copies of your email

Step 2: —In the second email, make one change

Step 3: —Send emails to two small selections of your targeted members

Sample breakdown:

Version A – 10% (ex. 100 of 1,000)
Version B – 10% (ex. 100 of 1,000)
Winning version – 80% (ex. 800 of 1,000)

This and step 4 may be the most difficult, depending on what kind of a system you are using for your emails. Some systems have A/B Testing functionality included, like the Databank, and others you need to create small samples yourself.

Step 4: Look at the results of the two emails to see which one performed better

—Step 5: Send winning version to the rest of your audience. In the end, 90% of your recipients will get the winning version

Measuring results

Once you’ve sent your A/B Tests and you’re looking at the results of them, how do you know what qualifies as a “winner”?

The most important step in measuring your results is to know your baseline. In other words, know what kind of results you were getting before you sent the email. What were your average open rates, click-throughs, etc. for your emails. This is important to know in order to compare how your A/B Tests are doing.

There are a few different ways you can measure the results, and that is by:

• Open rates – best to track if you are testing different subject lines
• Click-throughs – best to track if you are testing content/image changes
• Actions Taken – best to track if you are testing content/image changes on a call to action

Remember to track how the emails do in comparison to your normal baseline. Whichever version has the most increase is the one that will be crowned your “winner”.

Do you have any additional A/B Testing tips to offer?