A/B testing in email campaigns: how to improve ROI

Running email campaigns often feels like guesswork. You send a newsletter with a catchy subject line, but the open rate is disappointing. You try adding a discount code, yet sales barely move. The problem is simple: without a clear way to compare options, you never know what actually drives results. That is where A/B testing comes in.

It has one clear purpose: it helps you stop guessing. When you send two versions of an email to small groups and measure which one performs better, you learn what actually works with your audience.

What is A/B testing

A/B testing is a simple experiment. You send two versions of the same email, each with just one difference, to see which performs better. That difference could be the subject line, the call-to-action button, or even the time of day the email is sent. By comparing the results, you stop relying on opinions and start making decisions based on data.

The mechanics are straightforward:

1. First, you split your audience randomly into groups so the results aren’t biased.
2. One group receives the original version of your email, and another group gets the version with a single change.
3. After the send, you measure the chosen key metric, often called a KPI. For example, if you are testing subject lines, the KPI might be the open rate. If you are testing call-to-action buttons, the KPI might be the click-through rate. Once the data is in, you declare the winner and then send the better version to the rest of your list.

In practice, this could look as simple as testing “Spring sale starts now” against “Get 20% off today only” as subject lines. You send each version to a few hundred subscribers, see which one is opened more, and then roll out the winning subject line to the rest.

Types of A/B testing and when to use each

There are several ways to run experiments in email campaigns. The right choice depends on the size of your audience, the resources you have, and the type of decision you need to make.

— Classic A/B test

You compare just two versions, such as one subject line against another. This works best for smaller email lists, because you only need enough subscribers to get a clear result from two options.

— A/B/n test

When you add more versions, it becomes an A/B/n test. For example, instead of two subject lines, you test four. The upside is that you learn faster which angle resonates best. The downside is that each version gets fewer people, so you need a much larger list to get meaningful results. If you have only a few thousand subscribers, stick to two options. If you have tens of thousands, multiple versions make sense.

— Split test

Instead of changing a single element, you compare entirely different campaigns. For instance, one group might get a short, plain-text email while another group receives a longer, image-heavy design. Another common split test is experimenting with send times, like sending one version in the morning and the other in the evening. These tests are useful when you cannot isolate one element and want to see which overall strategy works better.

— Multivariate testing

This method changes several elements at once to see how they interact. For example, you test subject lines and button colors together. Multivariate testing can reveal how combinations influence results, but it requires a very large audience and clear hypotheses. Without enough traffic, the data becomes noisy and hard to trust.

Run an A/B test: step-by-step

Running an A/B test sounds technical, but in practice it’s just a sequence of common-sense steps. Each step keeps you from wasting time on experiments that don’t move the business.

1. Define your business objective

Testing should never happen in a vacuum. You need to know what you’re aiming for: more demo bookings, higher revenue per customer, more qualified leads, or improved retention. Without a clear goal, you risk “winning” a test that makes no difference to the bottom line.

2. Once you know the goal, you form a hypothesis

This is a short statement of what you expect to change and why. For instance, “If we use a shorter subject line that highlights the discount, open rates will increase because the offer is clearer.” A test without a hypothesis is just guesswork.

3. Choose one variable to test and pick the right KPI

If you test subject lines, your KPI is opens. If you test the CTA button, it’s clicks. If you test pricing copy, it’s conversions. Mixing several variables at once makes it impossible to know what actually caused the change.

4. Think about sample size and split

A simple rule of thumb is that each version should go to at least a few hundred recipients before you look at results. If your list is small, stick to simple A/B tests with just two versions. Split the audience randomly and evenly so each group is comparable.

5. Timing matters too

Send both versions at the same time to avoid bias. If one email goes out in the morning and the other in the evening, differences in performance may come from timing rather than content.

6. Let the test run long enough

Stopping early can give you misleading results. A good practice is to let the test collect responses for at least a few days, or until you’ve hit the minimum sample size. When the test is complete, declare the winner using the rules you set in advance. If version A beats version B on the KPI you defined, roll out the winner to the rest of your audience.

7. Document the result

Write down what you tested, what you expected, what happened, and what you’ll try next. This builds a record of insights that compounds over months.

For teams, it helps to have a short checklist before hitting send:

1. Did we link the test to a clear business objective?
2. Did we write a hypothesis?
3. Are we testing one variable at a time?
4. Do we know the KPI and minimum sample size?
5. Are both versions being sent simultaneously?
6. Have we set the duration and rules for declaring a winner?

How to interpret test results

Running an A/B test is only half the job. The harder part is reading the results in a way that actually guides decisions. Many teams fall into the trap of declaring a “winner” after looking at a single number, and then wonder why the improvement doesn’t hold in the long run.

— Do not stop at one metric

Think of the email funnel as a chain: opens, then clicks, then conversions. A subject line might give you a higher open rate, but if people open the email and don’t click through, the gain is meaningless. The only way to know if a change is worth rolling out is to follow the full cascade and check if it moves the business metric that matters.

— Check how different segments react

A version that boosts engagement among new subscribers might lower response among long-time customers. If you only look at the total average, you miss these nuances. A good practice is to glance at the results for major groups—like new versus returning leads—before making a final decision.

— Check size of the lift

A two percent increase in clicks sounds small, but if your campaigns already reach hundreds of thousands of people, that change can mean thousands of extra visits or sales each month.

— Be cautious with novelty effects

A fresh subject line or unusual design can create a temporary spike in performance simply because it looks different. The excitement may wear off after a few sends, so it’s wise to repeat the test or monitor the next campaigns to confirm the result holds.

FAQ

How long should a test run?

For most email campaigns, give it at least 24 hours. This covers different time zones and user habits. If you’re testing ads or web pages, you often need several days to reach enough people for the data to be reliable.

A good rule: stop only when both versions have had enough traffic to make the comparison meaningful. Ending early because one version looks ahead in the first few hours is a common mistake.

Can I test multiple elements at once?

Stick to one variable per test if you want clear answers. If you change the subject line and the sender name at the same time, you won’t know what actually drove the difference.

There’s an exception: if you’re using AI tools or running a multivariate test with big audiences, you can compare more elements. But for most small teams, one clear change at a time is the safest route.

What KPI should I trust — opens or clicks?

Opens are less reliable today because of privacy settings in email clients. They can still give directional insight, but clicks tell you more. A click means the person actually engaged, and conversions — like a purchase or a demo booking — are the most important signal of all.

Always follow the chain: opens lead to clicks, clicks lead to conversions. If the first number looks great but the last one is flat, the test hasn’t truly delivered.

How many tests per month is too many?

For small teams, one or two well-planned tests a month is enough. The goal isn’t to flood your audience with experiments, but to build steady learning over time. Larger companies with big audiences can run several in parallel, but each one should still be designed carefully and tracked. Running too many sloppy tests is worse than running fewer solid ones.

What if I don’t have a big list or traffic?

You can still test, but keep your expectations realistic. Instead of splitting into tiny groups where the result is noisy, run longer tests or focus on bigger changes. For example, if you only have a few hundred subscribers, don’t test two nearly identical subject lines. Test a short versus a long one, or a plain-text design versus a visual one. Bigger contrasts give clearer answers.

What if my test shows no difference at all?

That’s still useful. A “tie” means the change didn’t matter enough for your audience. It saves you from wasting energy on tweaks that don’t move the needle. Log the result and move on to the next idea. Over time, these learnings add up to a clear picture of what does and doesn’t influence behavior.

Can AI write my variants for me?

Yes, and it’s becoming common. But always test AI-generated content the same way you would human-written copy. Sometimes it produces bland text that looks fine but doesn’t resonate. Keep a human eye on tone, brand fit, and clarity. Use AI for speed and idea generation, not as a blind replacement.