A/B Testing: Running Experiments That Give Real Answers

What Testing Is Actually For

Testing exists to settle disagreements that opinion cannot settle. That sounds obvious, but it rules out most of what gets tested. If everyone in the room already agrees the change is an improvement, ship it. If the change is required for legal or accessibility reasons, ship it. Testing is for the cases where two plausible beliefs about customer behaviour lead to different designs and nobody can prove which is right.

The second thing testing is for is calibration. Teams that run experiments regularly develop a much more accurate sense of how much any single change is likely to move a number — which is usually far less than the case for doing it suggested. That recalibration is worth as much as any individual winning variant.

A Hypothesis Worth Testing

A weak hypothesis sounds like this: we think a green button will perform better. A strong one names the belief, the mechanism and the expected direction: we believe visitors hesitate at checkout because delivery cost is unknown until the final step, so showing it earlier will reduce drop-off between cart and payment.

The difference matters because the second version tells you what you have learned when the test fails. If the delivery hypothesis loses, you have learned something about your customers. If the green button loses, you have learned nothing you can carry to the next decision.

Where the Hypothesis Should Come From

The best hypotheses come from evidence you already have and have not looked at properly: support tickets, sales objections, session recordings, the search queries people type on your site and get nothing for. Ideas generated in a meeting from nothing tend to be a list of tweaks. Ideas generated from complaints tend to be a list of problems, which is far more useful.

Traffic, Sample Size and Honest Expectations

This is where most testing programmes quietly fail. Detecting a small difference reliably takes a lot of traffic, and the smaller the difference you want to detect, the more you need — the relationship is unforgiving. A site with modest traffic cannot resolve a two percent improvement in any reasonable timeframe, no matter how patient the team is.

The practical implication is liberating rather than depressing. If you do not have the traffic to detect small changes, stop testing small changes. Test the big ones: a different offer, a restructured page, a fundamentally different explanation of what you sell. Those move numbers far enough to be visible, and they are the decisions worth being right about anyway.

How Long to Run It

Decide the duration before you start, and make it a whole number of weeks. Behaviour differs between weekdays and weekends and between paydays and the rest of the month; a test that ends on a Thursday afternoon has sampled a shape, not an average. If a campaign, a holiday or a press mention lands mid-test, note it, because it will have brought in a different audience than the one you were measuring.

What Counts as the Same Visitor

Before the result means anything, check that the split is doing what you think. A visitor who sees variant A on their phone and variant B on their laptop is polluting both sides of the test. So is a returning customer who was assigned months ago under a previous version of the page. Confirm how assignment is stored, how long it persists, and what happens when someone clears their browser.

Check the split itself too. If the traffic is not landing close to even, something is wrong with the implementation rather than with your customers, and no amount of statistical care downstream will fix a broken allocation upstream.

The Three Ways Teams Fool Themselves

The first is peeking. Checking a running test every morning and stopping the moment it looks significant is the single most common way to manufacture a false winner, because random fluctuation will cross any threshold eventually if you keep looking. Set the end date and honour it.

The second is segment mining. If the overall result is flat, it is tempting to slice the data until some subgroup shows a win — mobile users in one country on a particular browser. With enough slices you will always find one. Unless you predicted that segment in advance, treat it as a hypothesis for the next test rather than a finding from this one.

The third is running several tests on the same page at once and then arguing about which one caused the change. If you must run overlapping tests, keep them on unrelated parts of the funnel and accept that the interpretation will be muddier.

What to Do With an Inconclusive Result

Most tests do not produce a clear winner, and that is normal rather than a failure. The useful default is to ship the version that is simpler to maintain, easier to explain, or better for accessibility, and record that the change did not measurably matter. That record is valuable: it stops the same idea being proposed again next year as though it were new.

A flat result is also a signal about where you are looking. If several tests on the same page all come back flat, the page is probably not the constraint. The problem is more likely upstream in traffic quality or in the offer itself, which is the territory covered in our guide to conversion rate optimisation.

Write the Decision Down Before You Start

Agree in advance what each outcome will cause you to do. If the variant wins, do you ship it everywhere, including the pages nobody tested? If it loses, does the original stay permanently or does the underlying question stay open? Writing this down takes five minutes and removes the most common post-test failure mode, which is a team that argues about interpretation because nobody agreed the rules while they were still neutral.

Keep a simple log of every test: the hypothesis, the dates, the result and the decision. Within a year it becomes the most valuable document your marketing team owns, because it is the only honest record of what your customers actually respond to rather than what successive people believed they would.

When Not to Test at All

Qualitative research is often faster and cheaper than an experiment. Five people trying to complete a purchase while you watch will tell you in an afternoon what a month-long test would tell you less clearly, and it will tell you why rather than only whether. Use testing to confirm a decision at scale, and use research to find out what to test.

There is also a measurement prerequisite. If you cannot trust the numbers underneath the test, the test result is not trustworthy either — a point we make in more detail in website analytics that answer business questions. Getting the definition of a conversion agreed and the tracking clean is not preparation for testing; it is most of the work. If you want that programme designed and run properly, our performance marketing team builds the measurement first and the experiment second, and treats the same discipline as part of ongoing campaign management.