Cirrus Solutions
Cirrus Solutions

Blog: A/B Testing for Marketing Success

Editor’s Note: This blog is part of a monthly series of marketing advice by Cirrus Solutions. To read more tips and tricks from the series, click here and here.

Doing A/B tests can help optimize your marketing performance by finding winning, elemental variants between two marketing campaigns. Sometimes called “split testing,” A/B testing is a method of assessing if potential revisions to your marketing should be rolled out.

This is done by testing the results of changing one variable against an otherwise identical marketing piece. Very small changes in those marketing elements can make a big difference in performance.

Sound complex? Not really. Here is all you need to run an A/B test:

What do you want to test?

The first thing to do is decide what you want to revise in your marketing. Consider what change(s) in behavior you would like improved because of your revisions.

For instance, let’s say you want to improve the open rates of your email newsletter. In this case, the subject line would have the most immediate effect. The recipient browses their inbox and the subject line either motivates them to open your email or not. 

Determine the KPIs and Sample Size

Determining which key performance indicators (KPIs) to measure should be part of the planning process.

In our newsletter example, your KPI would be open rates. Your sample size should be adequate to validate the results. For example, in most cases, 10 sales would not provide enough data to feel confident about the test results.

In the subject line example, this could be a quantity of recipients or a percent of the email list. Most of the email service providers (ESPs) include this type of testing in their standard platforms.

Once you’ve determined your sample size, divide your list into randomized groups (for example, Group A and Group B). Next, assign each subject line to be sent to a specific group.

This means all recipients would receive the exact same newsletter. The only variant is the subject line.

It is important that only one element of the execution be different to isolate the dependent variable. While you can do multi-variate tests, this is a more advanced concept and takes larger databases and analytics capability.

Analyze the Test Results

In our newsletter example, you will need to make sure there is a way to measure open rates for each group before you begin testing. You would not want to run a test and then not be able to see results.

Depending on your resources and level of customer tracking, you may be able to read further down the funnel beyond open rates. For instance, perhaps you can also see which email subject line resulted in an increase response to your call to action or higher sales.

As you run more A/B tests over time, it will be helpful to create a way to see a history of your past tests showing the winner, loser, samples and any relevant notes or information about each test.

This will help you understand how your marketing evolves through testing as well as help you spot additional opportunities to test. This can also help you quantify the impact, in real revenue, of testing.

Depending on the resources available to track results, A/B tests can be run on just about any media type or process.

Digital media is often the easiest to read results on. There are many popular tools that track digital activity 

However, as long as you are able to create a unique identifier to help you track results, it’s possible to A/B test just about any type of media – this includes, print, mail, broadcast, etc.

Depending on how granular your customer tracking is, you may be able to test whether a change deeper in your customer funnel can lead to improvements. For example, can different financing terms increase sales or average ticket?

Considerations of A/B tests

There are some things to consider before running an A/B test. 

A/B tests are ideal if you’re looking for longterm success. This is because you will need to wait to roll out changes until you can read your test results. Depending on the amount of data you need to collect before validating, this could be a day, a few weeks or even longer.

Be mindful of changes that you’d like to test. The portion of your test group that receives the losing version of the test, won’t convert as well as the group with the winning version. There’s a cost to this.

So, now that you’re ready to start A/B testing, here are some test ideas to help get the wheels turning:

  • Promotional offers
  • Subject line tests
  • Time of day to send emails
  • Lead form fields, copy, layout, etc.
  • Colors used
  • Placement of images or logo
  • Using one image versus using another image
  • Font type
  • Tag line updates
  • Call to action buttons: color, size, text, placement, etc.
  • Size of marketing piece
  • And more!


Yil Acosta is client relations manager at Cirrus Solutions, a dealer management system and marketing program provider built by dealers for dealers. For more information, visit

Got a news tip? Contact David MacNeal