<img height="1" width="1" style="display:none" src="https://www.facebook.com/tr?id=1822615684631785&amp;ev=PageView&amp;noscript=1"/>

Introducing A/B Testing

introducing AB testing

You can now A/B test your email outreach campaign using our powerful platform. Here are the basics:

  • Easily vary your Subject/Message with different sets of content.
  • Get separate open/click/reply statistics on each variation that’s sent.
  • Send to a portion of your list, and then decide which variation is the best, and then send the “winner” the rest of your list.
  • The “winner” can be chosen automatically by our algorithm or by you.

Just curious about using spintax to vary up your emails, with or without A/B testing involved? Jump to the spintax section of this article.

A demo

Animation of A/B test
I’m A/B testing my subject and body. I set the test so that I pick the winner manually.

A simple A/B test

To conduct an A/B test on a campaign, there are two steps to take:

  1. Create a variation using our “spin” syntax (aka spintax) in the subject or the message. Here’s an example:
    variation using spin syntax
  2. Apply the A/B settings in the settings panel. Here’s an example:
    apply AB settings in settings panel

In the above simple example, the subject line will be varied as the email begins to send. If I send to a list of 100, only 30 emails will be sent at first, with approximately 10 emails with the subject line “Hi”, 10 emails with the subject of “Hello”, and 10 with “Hey”. Four hours after the 30 emails are sent, a winner will be chosen automatically based on the open rate of all three segments. At that time, the remaining 70 emails will send with the winning subject line. Technically, this is an A/B/C test, since there are three variations.

This is what my Sent folder looks like while running the test.

A more sophisticated test

In a real life scenario, you’ll want to do more than just vary the subject line between Hi, Hello, and Hey. If you’re sending cold email, you may want to vary your introductory line. Here’s a more practical scenario:

spin variation - introductory line

Here, three different introductory lines will be used, and if I’m a cold emailer, I’m likely to care about the “reply” rate more than anything else. Because it can take a while for people to reply, I want to wait 24 hours before picking a winner, so my A/B test settings would look like this:

time setting for AB testing

Using A/B testing on a complete cold email sequence

You can A/B test your initial message plus each message in your email sequence. Here’s how that might look in the Settings box.

settings to AB test a follow-up series

Note that you don’t have to have the same number of variations in your follow-up template as you do in the original message. Each follow-up template can have a different number of variations in the message, but of course, this will make it more difficult to assess which version is really having the most impact.

Using the “manual” option

If I choose the “manual” option, I’ll be notified in my dashboard and via email when it’s time to pick a winner.

After the initial batch of emails are sent, and after the specified number of hours have passed, you’ll get an email that looks like this:

A/B notifier - time to choose a winner

You’ll also be alerted in your dashboard that it’s time to pick a winner.

You can then choose the winner from either the email notification or your dashboard, and then the remaining emails will be sent.

Personalizing inside the variations

You can use our standard personalization syntax inside the variation blocks. Using our Hi/Hey/Hello example, you can also do this:

{{spin}}Hi {FirstName}{{variation}}Hello {FirstName}{{variation}}Hey {FirstName}{{end spin}}

In fact, inside each variation you can use our full personalization syntax and our full conditional content scripting language.

Interested in statistics only?

You might want to conduct an A/B test but not have any decisions made, manually or automatically. Meaning, you might want to send to 100% of your list while varying your content. To do so, just set the slider in the A/B settings to 100%.

If you set the A/B settings to send to 100% of recipients, then:

  • All emails will go out at once, with an equal number of recipients receiving each variation
  • No decision will need to be made, automatically or manually
  • You can still see how each variation performs in your campaign report

Coming from a different A/B testing platform?

If you’re migrating from another email platform offering A/B testing, our way of doing things might be different and take some getting used to. Traditional email marketing platforms have you create an entirely different subject/message as your “B” version, while in GMass, you put in all the variations within one single message using our special spintax commands. We’ve always focused on efficient workflows, and we think our way is a faster, easier way to launch an A/B test.

In most A/B tests, some content of the message is similar between the variations, and with our way, you don’t have to duplicate the common content across message spaces.

Understanding line spacing when composing variations

When you put in a variation into the Message area, you’re using our conditional logic spin feature, consisting of these tags:

{{spin}}

{{variation}}

{{end spin}}

A line that has just a conditional logic code on it does not count as a line of space when our processor interprets your campaign.

So both of the following examples will produce the exact same line spacing:

{{spin}}
Hi there,
{{variation}}
Hello!
{{end spin}}

How are you doing?

and…

{{spin}}Hi there,{{variation}}Hello!{{end spin}}

How are you doing?

How to test your A/B variations

Before you conduct an A/B test campaign, you probably want to see what each variation of your full complete message will look like to the receiver.

The easiest way to test your variations is to use the “Send Test Email” button at the top of the settings box. When you send a test email to yourself, each possible variation of the email will be sent. For example, if my subject line looks like:

{{spin}}Hi{{variation}}Hello{{variation}}Hey{{end spin}} there!

Then when I send a test email, three test emails will be sent, each one showing the different subject line.

Additionally, you can also use the A/B feature along with the “Create Drafts” setting. By doing so, you can review the DRAFTs that are generated for each of your email addresses before sending. You can also then modify the contents of individual DRAFTs should that suit you. However, note that the “timer” for when the testing period expires starts after the DRAFTs are created, so make sure to get them sent relatively soon after they’re generated.

FAQ

After the initial test is sent, can I change my mind and send more as part of the test?

Yes! Just adjust the percentage slider up, and save your campaign. More emails will then be sent as part of the test.

What happens if I use the “spin” syntax (spintax) but don’t put in any A/B settings?

Then the email will still vary itself across the whole list, but a winner will never be chosen. Your entire list will be split amongst the variations.

What happens if I put in the A/B settings but don’t include any spintax in my Subject/Message?

You’ll get an error and won’t be able to launch your campaign, since you can’t conduct an A/B test without at least one variation.

Do the number of variations throughout the Subject and Message have to be equal?

Yes. For example, you cannot have 3 variations of your introductory text, and then just 2 variations of your second paragraph. That would produce inconsistent and confusing results, so we don’t allow it.

Spintax in GMass

A/B testing in GMass makes use of spintax. That is: The syntax you can use to send different variations of a word, sentence, paragraph, or other email element to different recipients.

And while the “traditional” use of spintax is for A/B testing, you can also use spintax if you just want to vary up your email copy — no testing involved. (Some cold emailers like this strategy, as they like as much variation as possible in the mass emails they send.)

As we’ve covered throughout this article, you can add spintax to a campaign in GMass by employing the three spintax tags: {{spin}}, {{variation}}, and {{end spin}}.

Spintax in an email message with GMass

For example, I could send an email where I’m spinning the way I introduce myself.

{{spin}}

My name is Ajay, and I live in Ohio.

{{variation}}

I’m Ajay, and I live in the Midwest.

{{end spin}}

I can send a test to myself — GMass automatically sends a test with each spintax option — to see the results. When I send the campaign, half of my list will receive one version, half will receive the other. (If I’m running an A/B test, it could be a different percentage. For example, if I want to test on 40% of my list before sending the winner, 20% will get one variation, 20% will get the other, then the remaining 60% will only get the better-performing version.)

Spintax in action

Spintax in the subject line with GMass

You can also use spintax in subject lines, with or without an A/B test.

For example: {{spin}}Want to make $1 million?{{variation}}Want to make $1 billion?{{variation}}Want to make $5?{{end spin}}

My emails would now go out with one-third of my list getting each of the variations.

Spintax in the subject line

Using spintax in multiple spots with GMass

You can also use spintax multiple times in an email. GMass won’t send out every possible permutation. It will send all of the first spin options together as one message, all of the second options together as one message, and so on. That’s why your different spintax sections all need to have the same amount of variants.

Here’s an email where I’ve used spintax in the subject line and twice in the email message.

Three spintax areas

And here are the results.

Three spins for two emails

So, again, you can use spintax for your A/B testing or in the emails you’re not A/B testing if you want variety. It’s as simple as using the three spintax tags and typing in your variations.

Ready to transform Gmail into an email marketing/cold email/mail merge tool?


Only GMass packs every email app into one tool — and brings it all into Gmail for you. Better emails. Tons of power. Easy to use.


TRY GMASS FOR FREE

Download Chrome extension - 30 second install!
No credit card required
Love what you're reading? Get the latest email strategy and tips & stay in touch.
   


19 Comments
  1. Hi Ajay,

    First of all; thank you a gazillion times for introducing this feature with Gmass. It will now power up our cold email campaigns and make them more effective like never before.

    I have a question though. Up to how many variations can I put in one cold email campaign? Mostly, i’ve seen people using three, but if my marketing team is more creative in writing, can they experiment upto 8 or 10 variations within your spin syntax ?

    Please advise

    Regards
    Ajay Thakar

  2. Looks interesting, can’t wait to try it.

    Is there a typo/mistake in the first example (under “A simple A/B test”)? It seems there are 3 possible subject lines (Hi, Hello, Hey) in the screenshot, but the text below says 20 emails will be sent, 10 with “Hi” and 10 with “Hey”. Or am I missing something?

    1. No. Open and click tracking are completely unrelated to this A/B testing feature. Although you just gave me a good idea — letting users do an A/B test with tracking on vs off.

  3. I’m trying to use If/Then/Else statements inside spin/variation but mys statements seems to be ignored or generate errors.
    Is it possible to use conditions inside A/B tests variations?

  4. It looks like the A/B test results has disappeared from the gmass dashboard (when I go into a campaign to look at the results, it is not broken out for A vs B and it used to be). Any suggestions?

  5. It seems that I can’t test different subject lines with A/B testing. Can one only test slightly different words like hi or hey, instead of different subjects? I receive this message:

    “Uh oh, something went wrong while sending. Here is the detailed error:

    General Exception This campaign cannot be A/B tested because the number of variations in each spin set is different.”

  6. I really really really wish you made all of your examples above copy/paste rather than images that can’t be copied. It would be way more helpful and allow those of us trying this out to have some sample text to not screw up with going back and forth to look at it and type it.

  7. Hey Ajay, I set up an A/B test for 2 SLs and 2 email messages. After sending and navigating to the live dashboard to see results come in, I noticed opens and clicks were being wrongly attributed to the wrong group. Since I sent to other emails of mine I know which groups I opened, clicked on, etc. And the results were reversed in the dashboard.

    Scary to see before possible using this feature with my email list.

    Any idea why this might be?

  8. For A/B testing, can I include with/without graphic as an alternative? That is, I would like to test an email which includes graphics and one without. Do I just treat the graphical elements as one may a text alternative?

  9. What happens if you make a massive campaign with 25,000 emails that is sending out 2000 per day? Will the a/b test pick a winner for all 25k or just that day’s 2k?

  10. Thank you for this very helpful tool and tutorial! I am curious about how to A/B test images in your email body content. We know people have strong responses to visuals. How would the spintax work to vary the embedded images? Is this an option? Thanks for your attention.

Leave a Reply

Your email address will not be published. Required fields are marked *

Find out why GMass is the most popular Gmail mail merge tool in the world

Easy to learn. Easy to use. Packed with time saving, email enhancing features.

Try GMass for free Then check out the quickstart guide to send your first mail merge email in minutes!

GMass

Share This