A/B test everything!
At Chapman Bright, we like to talk a lot about best practices. And while these are always a good starting point, you should also remember that every industry and every company will have different experiences.
And that is why we are also frequently recommending to our clients to A/B test their emails and landing pages. This article provides a quick overview of the options for A/B testing using Marketo Engage.
Test one idea at a time
As a quick reminder for those new to A/B tests, you will only get useful results if you are testing a single change. For example, if you are testing the effectiveness of a button color change, your button color can be the only thing that is different between your two emails. If you also tweak the location of the button…or the text on the button, then the A/B test will not give you useful data for future improvements.
What can you A/B test in emails?
There are four different types of A/B tests you can run within email programs in Marketo. Here are a few options for things you can A/B test in each category.
- Subject line
- Does including the recipient’s {{lead.First Name}} in the subject line increase open rates?
- Do shorter or longer subject lines perform better?
- Does including your business name in the subject line make a difference?
- Does including emojis make a difference?
- Whole emails
- Does personalization improve click-through rates? By including names, company name, etc.
- Do shorter emails perform better?
- Does this image banner get more clicks than that image banner? Does this color button get more clicks than that color?
- From Address
- Does sending from a “communications@company.com” email address perform better than a “marketing@company.com” email address?
- Does sending from a person’s name increase your open rate?
- Does it matter who that person is? Does sending from an account manager or sales person get even better open rates? How about the company CEO?
- Date/Time
- Do emails on Mondays perform better than those sent on Thursday?
- Does my target audience work unusual hours? Maybe 4:37AM and 7:09PM are better times to test.
- Does it make a difference if you send at exactly 10:00AM versus 10:04AM?
Adding an A/B test to your emails in Marketo is really easy. Within the control panel of your email program, simply click on the “Add A/B Test” button and follow the directions with the pop-up window. You can also add an A/B test to trigger and batch emails, but it is a little different.
What can you A/B test on landing pages?
In Marketo, there is only one technical requirement for setting up an A/B test for landing pages, which is having two or more different landing pages. How you make them different is completely up to you. And there is only one measurement of success, which is the conversion rate of how many people filled out the form.
Here are a few options for things you can A/B test on your landing pages:
- Images/Video
- Does having a video on the page increase conversion? Does having the video at the top or the bottom of the page make a difference?
- Does including an image or screenshot of the asset increase conversion? How about a photo of the author/presenter?
- Does having custom imagery increase conversion? In comparison to a stock photo?
- Text
- Does changing your landing page text increase conversion?
- Does making the text longer or shorter increase conversion?
- Does changing your primary call to action increase conversion?
- Forms
- Does reducing the number of fields increase conversion?
- Does making fewer fields mandatory increase conversion?
To add an A/B test to your landing pages, you will first need to create your two (or more) landing pages. While the pages are still unapproved, right click on one of them and click “Convert to Test Group”. In your new A/B test, click on the “Test Group Actions” link and select “Edit Group Members.” You can then find your other unapproved landing page to add it to the test group. After both pages are added to the A/B test, you can approve the pages and it will start measuring results.
Don’t forget about statistical significance
Maybe one of your pages saw a 100% higher conversion rate than your other landing page. That’s great! Except, maybe you only had three conversions total…
As you analyze your results, be sure to look at your volume to determine if your results are statistically significant. Results are considered statistically significant if there is little likelihood that the variation between A and B occurred by chance.
One way to ensure you have enough volume is to organize your A/B tests into themes over time. Start with all landing pages for three months (or six months) testing the same idea. While the landing pages will all be different and the videos will all be different, you can consistently test if adding a video makes a difference in conversion rates.
These results can then feed into your best practices and be implemented as changes within your example programs. And then you can start testing the next idea.