A/B Testing: Definition, How it Works, Examples & Tools

Types of testing

A/B testing is a CRO technique in its own right, but it comes in a few varieties. Depending on the information you need, you might want to use one of these types of A/B testing to get better insights into your shoppers.

Less than 50% of businesses use split testing, but it’s one of the most powerful tools you have to prove that your approach is effective.

Split testing

With split testing, you’re able to make data-driven decisions. Because you only test one element at a time, you can prove that a single change encourages more audience engagement.

For example, an A/B test normally would direct website traffic to the same URL but show a different design to visitors. Split URL testing works similarly, except it displays a page with a different URL to your users. This is a back-end change where you adjust the URL for each variation.

You might direct 50% of your traffic to www.example.com/freebies and the other 50% to www.example.com/resources. The unique URLs make it easier to monitor your A/B test results in an analytics tool.

Multivariate

In a typical A/B test, you’re testing a single variable across several variations. For example, if you want to test your website forms, you would create three versions of the form and keep the other variables on each page the same. This way, you’re comparing apples to apples every time.

Multivariate testing allows you to test multiple variables on multiple pages all at once. If you want to test CTAs, images, headers, and more, you don’t have to run separate A/B tests for each option — and that saves you a lot of time. All you need to do is run a multivariate test, and you’ll get all of your answers from one campaign.

However, multivariate testing is much more complex than a typical A/B test because you’re testing so many variables. For that reason, it’s best to wait to do multivariate testing until you have a few A/B tests under your belt first.

Multi-page

A/B testing typically looks at changes made to one aspect of your digital marketing. But that’s not representative of the entire customer journey. What shoppers see before they reach your website, app, or email can influence how they interact with your content. Sometimes you need to test the entire experience leading up to a landing page, and that’s where multi-page testing comes in.

With multi-page testing, you can test every stage of the funnel, giving shoppers a different version of your content to see what sticks. Multi-page testing is less myopic than A/B testing because it looks at the net effect of the customer journey (rather than a single page). However, it requires a lot of content creation and planning, so executing this type of test can take a lot of time.

Benefits of A/B testing

Although there are different types of testing available to your brand, A/B testing is still the best place to start. That’s because A/B testing helps you better understand your customers, quantify what works, improve ROI, increase time on site, and evaluate your efforts.

Give customers what they want.

Your business exists to solve customers’ biggest problems. But you need to know if you are. For example, maybe you think you should create a customer loyalty programme, but your shoppers want free delivery. You won’t know what engages your customers best unless you run a test first. In this way, split testing helps you to understand what motivates your shoppers to take action — and how you can convince them to choose your brand over everyone else.

Quantify what works

With A/B testing, there’s no guesswork. Your graphic designer doesn’t need to wonder which layouts or colours your audience likes. And your app developer knows exactly what type of experience your users expect from your branded app. A/B testing software allows you to run tests on a constant basis, generating helpful data that you can use to quantify what works. If you’re trying to run a more data-driven business, A/B testing is a must.

Improve ROI

You’re already investing in email, social media, your app, products, and a website. You want to get results from all of your hard work. A/B testing helps brands enjoy increased engagement and all the return on investment (ROI) that comes with it. 60% of businesses that do A/B testing say it’s highly valuable for increasing conversion rates. Furthermore, 70% of brands increased landing page sales with A/B testing, and some businesses reported as much as a 50% increase in average revenue per user.

Increase time on site.

The more time users spend on your site, the more interested they’ll be in your solution. If people bounce from your site within a few seconds, however, they aren’t interacting with your brand. That translates into lost conversions and fewer sales. But with split testing, you know which elements people want to see. You can give your audience exactly what they need to stay on your website for as long as possible.

Increased time on site is also a positive indicator to search engines. If you want to rank higher in search engines like Google, increasing your time on site can improve your organic search traffic.

Know what moves the needle.

Less than 50% of businesses use split testing, but it’s one of the most powerful tools you have to prove that your approach is effective. With A/B testing, you can learn what works for your business. Because you test just one variable at a time, you can demonstrate which changes are most impactful. That means you’ll spend less time on experiences that don’t work and focus your resources on what shoppers want to see.

How does A/B testing work?

Most marketers A/B test their landing pages, email, and pay-per-click (PPC) ads, but you can use it for just about anything.

For example, you can use A/B testing to discover how:

• Decreasing the number of form fields in the checkout process increases purchases.
• Adding testimonials to a product page increases conversions.
• Your email click-through rates increase when you add a video thumbnail to the email body copy.

Just 7% of brands say it’s hard to do A/B testing. So the good news is that with the right tools, split testing is a pretty easy process. Every organisation is different, but A/B testing usually follows these steps:

• Create a hypothesis. Proper A/B testing should follow the scientific method. To start your experiment, you need to have a theory. Your hypothesis may or may not work out, but it’s an assumption to test. For example, your hypothesis might be, “Making the CTA button orange will increase click-throughs” or “Adding emojis to our email subject lines will increase open rates.”
• Choose your versions and variables. Next, you’ll choose a variable that fits with your hypothesis and create two versions of it. All of the other elements of your email, page, or app should be identical — just one variable must be different. You’ll need to create at least two versions. Just keep in mind that tracking performance across a handful of versions might make it more difficult to understand your results.
• Use split-testing software. You can’t do A/B testing manually. You’ll need to use split-testing software to serve different versions of a page, app, or email to your audience. The software will measure how people engage with the different versions and report on the data. At the end of the experiment, you can see which version got the most positive results.

CTAs are the most common variable that brands split test. One-third of businesses that do A/B testing always test their CTAs.

What should you test?

With A/B testing, you can test just about any element of a page, app, or email. But here are some of the most important things to test.

Titles and headlines

Page titles and headlines give users an idea of what to expect from your content. Search engines also scan your titles and headlines to see what your content is about, so split testing could improve your SEO and ranking over time, too.

CTAs

Calls to action (CTAs) tell your audience what you want them to do next. Since CTAs help you transform leads into paying customers, it’s no wonder that one-third of brands that do A/B testing always test their CTAs. CTAs are the most common variable that brands split test.

If users aren’t biting on your current CTAs, split test a few options and see if you get better results. For example, you might see that “Learn more” is too weak a CTA.

Instead, substitute it for these stronger CTAs:

• Shop now
• Get the free guide
• Send message
• Subscribe now
• Join our mission
• Donate here
• Sign up
• Try for free
• Get started

There are so many CTAs you can try out, so drop in a few options to see what works. Just make sure the CTAs are relevant. If you don’t want people to “Learn more” and want them to start shopping, say that. Transparency builds trust, so be upfront about what the next step is.

Landing pages

You rely on landing pages to generate leads, bring in more sales, and promote your products. You put a lot of thought into your landing pages, so use A/B testing to measure their effectiveness.

The average landing page conversion rate is 26%. But with split testing, some businesses report seeing conversion rates over 70%. Even if you don’t convert 70% of your traffic, you can boost landing page performance by split testing these variables:

• Headline
• Benefits
• Social proof
• Multimedia like photos and videos
• CTAs

Images

The images you use on your website, app, landing pages, and emails tell users what they can expect from you. You can say a lot with a well-placed image, so A/B test your images to see what sticks.

That might mean:

• Trying out different colour palettes
• Testing lifestyle photos versus product photos
• Testing a male versus a female subject
• Testing realistic versus cartoon images

Whatever you do, make sure the images are always high-quality. Low-quality images will lose your shoppers’ trust and hurt your conversion rates. Your images should be at least 300 dots per inch (DPI).

Forms

Forms are necessary for collecting information from your customers, but a lot can go wrong. More than 80% of people who start to fill in a form will leave, but A/B testing can help you identify ways to improve your conversions.

You can test:

• The length of the form
• Adding or removing a progress bar
• Adding or removing example text in each form field
• The copy used on the “Submit” button
• Colour and design elements like font

Navigation

That’s right — you can even split test your navigation menu. This is especially helpful if you want to evaluate your home page or landing page performance. Try A/B testing:

• Where you place navigation elements
• Which items do you include in the navigation
• How many items are in the navigation menu
• Whether you should have a navigation menu at all

Email subject lines

You need solid subject lines if you want people to open your emails and take action. But you need to determine which subject lines persuade people to open an email — and which options will send you straight to their trash folder.

Aside from CTAs, email subject lines are some of the most common variables to split test. Nearly 60% of brands report split testing their email subject lines. Many email service providers (ESPs) offer this feature out of the box, so split testing subject lines is extremely easy.

Many brands adjust these variables to improve their email performance:

• Adding the recipient’s name or business name
• Asking questions
• Adding emojis
• Creating urgency (“Sale ends in two hours”) or scarcity (“Only two slots left”)

Copy

No matter how you slice it, body copy is the meat of your app, landing page, or email content. The good news is that A/B testing allows you to assess any piece of copy.

Split test your product descriptions, website home page taglines, and ad copy to see what resonates most with your audience. Play with the tone (whether people prefer casual or formal), formatting (whether people like headers, bullets, or bold text), length (whether they prefer longer or shorter copy), and more.

Page layout

A/B testing page layout is an option that often falls by the wayside, but how you lay out a page can have a big effect on the user experience. Split test where you place elements on your website and see if it makes a difference. Test things like:

• Social proof
• Carousel versus fixed images
• Long versus short pages
• Font sizes
• Colour schemes

How to do A/B testing

A/B testing isn’t complicated, but you do need to follow a careful strategy to do it correctly. You want to keep your data as clean as possible to make the best decisions. Follow these seven steps to conduct better A/B tests.

1. Test one element at a time

First things first — meet with your team and decide which elements you want to test. It’s tempting to create two unique landing pages and compare them against each other, but that will only make it harder to determine which variables had an impact on conversion rates. With traditional A/B testing, you only want to test one element at a time. This way, you know which changes led to positive (or negative) results.

2. Establish a goal

Once you decide on the variable you plan to test, get clear on your goals. What do you want to get out of this split test? Conversions are the most common objective, but your team might also want to:

• Increase form submissions
• Lower bounce rates
• Decrease abandoned baskets

Once you know your goal, you’ll need a baseline for comparison. Where are you right now? For example, if you want to lower bounce rates, you need to know what your current bounce rates are. This baseline will help you to prove that the changes you made benefited your brand. So if your bounce rate is 40% right now and your A/B test gets it down to 20%, that’s a sign that you’re on the right track.

3. Create a variant

Now it’s time to create a variant. If you’re new to A/B testing, stick with two variants to start. For example, if you want to compare CTAs, test two identical pages with different CTAs and see which one performs better.

We recommend testing two variants at first because adding variants makes the experiment more complex. It also makes it harder to find statistically significant results. If you’re new to A/B testing, stick with a control page and a variant for your first test.

4. Split sample groups, if necessary

This isn’t always necessary, but sometimes you’ll need to watch your audience demographics during a split test. For example, if you’re split testing emails, you need to make sure you send the split test to two similar audiences. If you send them to a segment of longtime customers and a segment of new leads, you’re going to have skewed results because these two audiences are so different.

If you only want to see how a certain segment reacts to your split test, ensure that your test is only reaching people in that audience.

5. Decide on a schedule

You shouldn’t run A/B tests in perpetuity. They’re an experiment, and all experiments need a start and end date so you can gather data and analyse the results.

Your split test needs to be long enough for you to gather meaningful data. Two weeks will usually give you enough time, but if you have a particularly large audience or a lot of web traffic, you may be able to test within a shorter window than that.

Smart split testing helps brands understand their customers, improve the digital experience, and stay one step ahead of industry trends.

6. Analyse and act on results

When the A/B test ends, sit down with your team to analyse the results. Your split testing software will pull all of the analytics for you so you can track the performance differences between your two versions.

It’s important to track clicks, shares, and other engagement metrics. But what matters most is whether you achieved the initial goal you set at the beginning of the A/B test.

It’s okay if you didn’t. Unexpected things occur in experiments all the time, so whether you met your goal or not, use this time to dig into the analytics and see what happened. A/B testing is a science, and science will often surprise you. Let the numbers speak for themselves — follow the data and make changes to your approach based on the research.

7. Rinse and repeat

A/B testing isn’t something you do on a one-off basis. Unfortunately, 57% of brands stop split testing once they get their desired results, but there’s so much more for you to uncover about your customers. A/B testing should be a regular part of your marketing team’s CRO strategy.

Everybody wants to see a lift in conversions across all of their campaigns, so don’t lose momentum after the first split test. From here, it’s a matter of rinsing and repeating. With more tests under your belt, you’ll gather even more data and have a better understanding of your audience’s needs.

How to analyse A/B testing data

A/B testing is critical for improving your conversion rates and your marketing approach as a whole. But it creates a lot of data for your team to sift through, so you need to know the best way to glean insights from that split test data.

If you’re using A/B testing software, it will do a lot of the calculations for you. Even then, your team needs to interpret what these calculations mean. You don’t need a statistician to do that, either. Follow these tips to find actionable insights from your split test experiments.

Sample size

If you gathered data from 1,000 visits to your website, you’re going to get lots of information. But if you only got 10 visits to your landing page, that isn’t a great sample size. If you have a small sample size, take your findings with a grain of salt. After all, with only 10 people in a sample, the actions of one person account for 10% of your findings, which will skew the data.

You don’t want to take action on incomplete or incorrect data, so be mindful of your sample size. If you didn’t get enough visitors this time, try to boost traffic for the next experiment so you get a bigger sample size.

Statistical significance

If you see a 1% difference between landing page An and landing page B, that likely isn’t statistically significant enough to say one version was better than the other. Generally speaking, small differences in performance mean that your data isn’t definitive enough to guide your decisions either way. At a minimum, you need at least a 5% difference in performance to determine which version is superior.

Goals

Ask yourself a few questions. What was your goal for this particular A/B test? Did you achieve it? Which variation led to the best results?

If you didn’t achieve your goal, that’s important information too. For example, maybe you thought changing the CTA button from black to red would increase conversions, but the opposite happened. You tested your assumption, and it didn’t make a difference, but that’s still significant — now you know that a black button performs better.

Segmentation

As long as you use a robust A/B testing platform, you can look at the split test data to see how it performed across your different audience segments. Look at the data as a whole and then break it down based on the following:

• Demographics
• Device
• Buying history

You might find that your email performed better for desktop users than mobile users. Or that a specific buyer persona was more likely to convert on a CTA than your other personas.

Check for anomalies or errors.

Nobody likes to admit it, but sometimes problems can happen in an experiment. Maybe you accidentally ran your campaign for too long — or not long enough. Perhaps you unintentionally tested a variable that you meant to save for a different split test.

These actions can muddy the waters and skew your data. Problems happen, but make sure your A/B test results are valid before you treat them as gospel.

Getting started with A/B testing

Smart split testing helps brands understand their customers, improve the digital experience, and stay one step ahead of industry trends. Adobe Target helps both enterprises and growing brands with better A/B testing that gets results.

Built for marketers, Adobe Target makes it easy to perform testing, measurement, and optimisation — all without writing code. With the power of unified customer profiles and machine learning from Adobe Sensei, Adobe Target personalises experiences based on historical behaviour rather than the latest transaction.

A/B testing might require some research, but it should be part of every data-driven organisation’s playbook — especially in these competitive times. When it’s time to start A/B testing, find the right tool for the job. From there, review which elements you want to test. It’s as easy as plugging the elements into your A/B testing platform, analysing the results, and making the changes going forward.