Testing two versions of an SMS campaign: a simple A/B test with contact groups

The short answer: an A/B test means sending two versions of the same campaign to two similar groups, changing only one thing, and seeing which brings more responses. In the SMS Center you do it with two contact groups and two campaigns. The winning version then goes to the rest of the list.

What to test

Change only this	Version A	Version B
The offer	[COMPANY]: 15% off everything until Sunday.	[COMPANY]: buy 3, pay for 2, until Sunday.
The customer's name	[COMPANY]: the new collection is in.	Hi {$contact_first_name}, the new [COMPANY] collection is in.
The deadline	Valid until Sunday.	Today only, until 8pm.
The time	Same message at 10am	Same message at 3pm
How to take it up	Show this SMS at the till.	Order at [WEBSITE].

If you change two things at once, you cannot tell which one made the difference.

Step by step

1	Export the group you want to test, under Contacts, in the Import and export tab.

2	Split the list at random into two halves in a spreadsheet, for example even rows on one side and odd rows on the other. Do not split alphabetically or by date: the groups need to be alike.

3	Import each half into a new group: "Test A" and "Test B".

4	Give each version its own signal: a different code (for example SUMMER-A and SUMMER-B) or a different UTM link. Without it you cannot count. See how to measure a campaign.

5	Create two campaigns, one per group, at the same time (unless time is what you are testing).

6	Count the responses to each version until the offer ends, and compare.

Test on a sample, then send the winner

With a big list, you do not need to split all of it. Take two equal samples, send A and B, wait for the result and send the winning version to the rest of the list. With a small list, split the whole thing in half and use what you learn in the next campaign.

How to read the result

Compare the response rate (responses ÷ messages sent), not the raw number of responses, if the groups are different sizes. With few numbers, a small difference can be chance: if version A got 12 responses and B got 10, run the test again before drawing conclusions. A big difference that repeats over two or three campaigns is a rule for your list.

Test length too. If the short version (1 part) does as well as the long one (2 parts), you use half the credits on every campaign from then on. See how many credits an SMS uses.

Both groups should only include people who agreed to receive offers. A test is no reason to send to new numbers or to people who opted out.

Want to find out which message works best with your customers? See SMS Marketing.

See the SMS Marketing page

Categories

Categories