Shopify Rollouts: what native A/B testing can and can't tell you

Shopify's Rollouts feature put split testing into the admin, server-side, with no app fee and no flicker. That is a genuinely good piece of infrastructure, and it has produced a predictable side effect: merchants who have never run a test before are now running them, reading the results dashboard as a scoreboard, and shipping changes based on numbers that cannot support the conclusion. Rollouts gives you the mechanism. It does not give you the statistics, and it does not tell you when your store is too small for the test you just started.

What Rollouts actually is

Rollouts lives in the Shopify admin under Markets, and is also reachable from Online Store, Themes next to your published theme. It does two related jobs that are worth keeping separate in your head.

The first is staged deployment: publish a new theme to a small share of traffic, watch that nothing catastrophic happens, then increase the share. That is a safety mechanism borrowed from software release engineering, and it is valuable on its own terms. The second is experimentation: split traffic between two versions to learn which performs better. Shopify ships both through the same interface, which is convenient and also the source of most of the confusion, because they demand completely different levels of rigour. A staged rollout is successful if nothing breaks. An experiment is successful only if the result is real.

The implementation details are good. It runs server-side, so there is none of the flicker that client-side testing scripts cause when the original version paints before the variant swaps in, a flicker that measurably depresses the variant it is meant to test. There is no app subscription. On plan availability, published guidance is inconsistent about exactly which tier unlocks experiments as opposed to scheduling and gradual rollouts, so check Settings, Plan in your own admin and Shopify's current documentation rather than trusting a third-party summary, including this one.

What Rollouts can and cannot test

The scope limit is the first thing to establish, because it determines whether the tool is relevant to the question you actually want answered.

The pattern: Rollouts is a theme testing tool that also covers checkout configuration. If your hypothesis is about presentation, it is the right tool and it is free. If your hypothesis is about pricing, promotions, personalisation or logic, it is the wrong tool and no amount of persistence will make it the right one.

The number Rollouts doesn't show you

Rollouts reports how each version performed. It does not tell you whether the difference is real. There is no documented significance test, no winner declaration, and no revenue-per-visitor figure, so a merchant looking at "Version B: 2.3%, Version A: 2.0%" has a 15% relative difference on screen and no information at all about whether that difference would survive running the test again.

That gap is not a criticism of Shopify. Every testing tool that does declare winners is making assumptions on your behalf, and a meaningful share of the "wins" merchants shipped over the last decade came from tools declaring significance too eagerly. But it does mean the statistical work is yours, and the first piece of that work happens before the test starts, not after.

How much traffic you actually need

The question to answer before launching any test is: what is the smallest improvement this test could reliably detect, given my traffic? The table below gives the sessions required per variant to detect a given relative lift, calculated with the standard two-proportion sample size formula at 95% confidence and 80% power.

Sessions required per variant. Two-sided test, 95% confidence, 80% power, using n = 2(z_α/2 + z_β)² p(1−p) / δ², where δ is the absolute difference in conversion rate. Double these figures for the total traffic needed across both variants.

Now put a real store through it. Take a shop doing 40,000 sessions a month at a 2% conversion rate, splitting traffic evenly, so 20,000 sessions per variant per month:

1. A 30% lift needs 8,500 sessions per variant: about 13 days. Detectable.
2. A 20% lift needs 19,200 per variant: about 29 days. Detectable, just.
3. A 10% lift needs 76,900 per variant: about 115 days. Not realistically detectable.
4. A 5% lift needs 307,700 per variant: about 15 months. Not detectable at all.

Most theme changes, a repositioned button, a different hero image, a reordered set of trust badges, produce lifts in the low single digits when they produce anything. A 40,000-session store cannot measure those. It can only measure changes big enough to be visible from orbit, which in practice means whole-page or whole-theme changes rather than element tweaks. That is not a reason to avoid testing. It is a reason to test bigger things and stop running four-week tests on button colours, and it is the same underlying reason most A/B tests fail to replicate.

Four ways Rollouts results go wrong

1. Reading a staged rollout as an experiment. A ramp from a small traffic share upward is not a randomised comparison over a matched period. The 10% cohort and the 100% cohort ran at different times, under different traffic mixes. Use ramps for deployment safety, and separate, fixed-split tests for learning.
2. Stopping when the number looks good. Conversion rate differences swing wildly early in a test and converge as sample accumulates. Checking daily and stopping on the first favourable reading is the single most reliable way to manufacture a fake win. Decide the end date and required sample before starting, then do not look for a verdict until you reach it.
3. Testing across a seasonal boundary. A test that needs four months to reach sample will run through Q4. Your baseline conversion rate, traffic mix and customer intent all change in that period, which breaks the comparison even though the split itself stayed even. Long test durations are not just slow; past a point they are invalid.
4. Ignoring who is in the sample. Automated traffic and non-human agents are a real and growing share of store sessions, and they enter both variants without converting, diluting the difference you are trying to measure. If your analytics and your Shopify order count disagree substantially, resolve that before you trust any test, and read up on how bot traffic distorts experiment results.

So do you still need a dedicated testing tool?

For most stores under roughly 100,000 monthly sessions: no, and buying one would be solving the wrong problem. At that traffic level your constraint is sample size, not tooling, and a paid platform will not manufacture visitors. Use Rollouts for whole-theme and whole-page comparisons, ship the small stuff on judgement and qualitative evidence, and put the subscription money into research that does not require statistical power, session recordings, customer interviews, checkout usability review.

A dedicated platform earns its cost when at least one of three things is true: you have the traffic to detect single-digit lifts; you need to test outside the theme layer, in pricing, promotions or logic; or you need audience-level targeting that Rollouts does not offer. Those are real needs, and they belong to larger stores. They are not a starting point.

Frequently asked questions

What is Shopify Rollouts?

Rollouts is Shopify's native feature for staged theme deployment and A/B testing, found in the admin under Markets and from Online Store, Themes. It splits traffic between your published theme and a copy server-side, with no flicker and no app subscription, and lets you ramp a change from a small traffic share up to 100%.

What can't you test with Shopify Rollouts?

Rollouts is scoped to theme editor changes plus checkout and customer account configuration. It cannot test product pricing, discount and promotion logic, custom checkout journeys beyond available settings, post-purchase flows, app embed behaviour, or deeper Liquid template logic. Audience-level targeting is also limited, so you cannot narrow a test to a specific visitor segment.

Does Rollouts tell you which version won?

No. It reports how each version performed but includes no documented significance test, no winner declaration, and no revenue-per-visitor metric. Deciding whether a difference is real, and whether your sample was large enough to answer that, is your responsibility, and it needs to be settled before the test starts.

How much traffic do you need to A/B test a Shopify store?

It depends on your baseline conversion rate and the size of the improvement you are looking for. At a 2% conversion rate, detecting a 20% relative lift needs about 19,200 sessions per variant at 95% confidence and 80% power; detecting a 10% lift needs about 76,900 per variant. A store with 40,000 monthly sessions can therefore resolve a 20% lift in about four weeks, but would need roughly four months for a 10% lift.

Is Rollouts free, and which Shopify plan do you need?

There is no add-on fee. Scheduling and gradual rollouts are available on lower plans, while running experiments and accessing the analytics requires a higher tier. Published third-party guides disagree on exactly which tier that is, so confirm against Settings, Plan in your admin and Shopify's own current documentation.

Should I still pay for a third-party A/B testing tool?

Only if you have enough traffic to detect single-digit lifts, need to test outside the theme layer such as pricing or promotion logic, or need audience-level targeting. Below roughly 100,000 monthly sessions the limiting factor is sample size rather than tooling, and the budget is better spent on qualitative research that does not require statistical power.

The takeaway

Rollouts removed the two things that used to stop merchants testing: cost and flicker. What it did not remove is the arithmetic. The most useful habit you can build around it is the least exciting one, before every test, calculate the smallest lift your traffic can actually resolve in a sensible timeframe, and refuse to run tests that fall below it. Most stores that adopt that rule end up running fewer, larger tests and shipping more confident changes, which is the correct outcome even though it looks like less activity.