Shopify tests themes natively now. Do you still need an A/B testing app?

Shopify's Rollouts feature turned a paid app category into a plan feature this year. Since the 5 June 2026 expansion you can schedule a theme, release it gradually, or run an A/B test between two entirely different themes or checkout configurations from inside the admin, with no third-party script involved. The obvious question is whether the testing app you are paying for is now redundant. The obvious answer is wrong, and for a large number of stores the real answer is that neither tool is the thing holding your conversion rate down.

What Rollouts is, and what it operates on

Rollouts first appeared at Shopify Editions Winter '26 in January 2026, reached the developer changelog on 31 March 2026, and expanded on 5 June 2026 to cover whole themes plus checkout and customer account configurations. Three things sit under one feature:

• Scheduling — publish a theme at a set time instead of at midnight by hand.
• Gradual publishing — release a new theme to a percentage of traffic and increase it, so a bad release is caught on 5% of sessions rather than all of them.
• A/B testing — split traffic between two configurations and compare how they perform.

Scheduling and gradual rollouts are available from the Basic plan upward. Running an A/B experiment requires the Grow plan or higher.

The important limitation is not on the feature list, it is in the noun. Rollouts works on published configurations. A theme, a checkout configuration, a customer account configuration. It does not operate on things that live outside those configurations, and it does not give you a way to change one element inside a theme without producing a second theme to hold the change.

The unit of change is the real difference

Price is the wrong axis to compare these on. What you can express as a test is the right one.

Read the table and the overlap is narrower than the headlines suggest. Rollouts is excellent at the thing testing apps have always been bad at — comparing two complete experiences safely — and cannot do the thing they exist for, which is changing one element without shipping a whole new theme.

Before you choose a tool, check you have the traffic

This is the part the tool comparison articles skip, and it decides the answer for most stores. A test only tells you something when it can distinguish a real difference from noise, and the sample size that requires is set by your conversion rate and the size of the lift you are hoping to see — not by which software you bought.

The table below assumes a 2% baseline conversion rate, two variants, 95% confidence and 80% power — the ordinary defaults. Numbers are rounded, and they are sessions, not visitors.

Now put that next to what theme-level changes actually produce. A better product-image gallery, a clearer delivery message, a reordered product page: these are 2–8% changes on a good day. A 30% lift from a theme tweak is rare enough that if your tool reports one on 17,000 sessions, the most likely explanation is that you stopped the test when it looked good.

So a store on 30,000 sessions a month can reliably detect only changes of roughly 20% or more within a single month. Everything smaller is invisible to it, no matter how good the software is. That is not an argument against caring about conversion rate. It is an argument that below a traffic threshold, testing is the wrong instrument — a point we made at length in why most A/B tests fail.

The cost comparison, which is smaller than you think

On 2026 pricing, Shopify Basic is $39 a month and Grow is $105 when billed monthly, or $29 and $79 billed annually. So if experiments are the only reason to move up, the step from Basic to Grow costs $66 a month, or $50 on annual billing — below the monthly price of most dedicated testing apps, and the Grow plan brings other things with it.

If you are already on Grow, Advanced or Plus, Rollouts experiments add nothing to your bill at all.

That makes the cost question easy and slightly beside the point. The money you are deciding about is not the licence. It is the time somebody spends designing variants, watching tests and interpreting them, which costs more than either tool and is the input most likely to be missing.

The hidden cost of a client-side testing script

Third-party testing tools work by loading a script that rewrites the page in the visitor's browser. That has two consequences worth pricing in.

The first is flicker: the original content paints, then the variant replaces it. Anti-flicker snippets hide the page until the script resolves, which trades a visible flash for a slower first paint. Either way the visitor pays.

The second is that the script is a permanent addition to every page load, including the pages with no test running on them. On a Shopify theme already carrying a stack of apps, one more render-blocking script on the product and cart path is a real conversion cost, and it is being paid continuously to support tests that run intermittently. We covered the size of that effect in Shopify theme speed and conversion.

Rollouts has neither problem, because the split happens before the page is served. For a store whose tests are infrequent, that difference alone can outweigh the app's greater flexibility.

Choosing

If you are below the threshold

Being unable to test is not the same as being unable to improve. The changes worth making at low traffic are the ones where the evidence already exists somewhere other than your own store:

1. Fix what session recordings and support tickets already tell you. Twenty recordings of people abandoning at the same step is better evidence than an underpowered test, and it costs nothing.
2. Ship the changes with published research behind them. Checkout field count, guest checkout, delivery-cost visibility before the final step: these are documented patterns, not hypotheses that need re-proving on your traffic.
3. Measure at a coarser grain. Before-and-after on a monthly cohort will not isolate causes, but it will catch a change that made things clearly worse, which is most of the protection a small store needs.
4. Use gradual rollouts as insurance. Releasing a new theme to 10% of traffic for a week catches the broken variant selector that would otherwise cost you a fortnight.
5. Put the effort into traffic and offer. At 30,000 sessions a month, a 20% lift in qualified traffic is usually easier to get, and worth more, than a 3% lift in conversion rate you cannot even measure.

That prioritisation is the same one we use when we start conversion optimisation work with a store: known fixes first, testing only once the traffic can carry it, and a rollout mechanism in place before either.

Frequently asked questions

What is Shopify Rollouts?

Rollouts is Shopify's native feature for scheduling a theme publish, releasing a theme gradually to a percentage of traffic, and A/B testing between published configurations. It first appeared at Shopify Editions Winter '26 in January 2026 and expanded on 5 June 2026 to cover whole themes plus checkout and customer account configurations. Scheduling and gradual rollouts are available from the Basic plan upward; running an A/B experiment requires the Grow plan or higher.

Does Shopify Rollouts replace a third-party A/B testing app?

Only for some roadmaps. Rollouts tests whole published configurations, so it is well suited to comparing two themes or two checkout configurations. It cannot change a single section, headline or block inside one theme without you building a second theme to hold that change, and it does not offer audience targeting. If your testing backlog is element-level hypotheses run several at a time, an app still does something Rollouts does not. If your roadmap is redesigns and configuration changes a few times a year, Rollouts replaces the app.

What plan do I need to run an A/B test on Shopify?

The Grow plan or higher. Scheduling and gradual rollouts work from Basic. On 2026 pricing, Basic is $39 a month and Grow is $105 when billed monthly, or $29 and $79 billed annually, so moving up purely for experiments costs about $66 a month, or $50 on annual billing. Stores already on Grow, Advanced or Plus get Rollouts experiments at no additional cost.

How much traffic do I need to A/B test a Shopify store?

More than most stores have. At a 2% conversion rate with 95% confidence and 80% power, detecting a 10% relative lift needs roughly 154,000 sessions in total, and detecting a 5% lift needs roughly 616,000. A 20% lift needs about 38,000. In practice a store on 30,000 sessions a month can only reliably detect changes of around 20% or more within a month, while realistic theme-level changes produce 2 to 8%. Below roughly 30,000 sessions a month, testing is the wrong instrument and known fixes are the better use of the budget.

Does an A/B testing app slow down a Shopify store?

Yes, to some degree, and the cost is continuous. Client-side testing tools load a script on every page, including pages with no active test, and either allow a visible flicker as the variant replaces the original or hide the page behind an anti-flicker snippet until the script resolves. On a theme already carrying several apps, that is a real conversion cost paid all the time to support tests that run intermittently. Rollouts avoids both problems because the split happens before the page is served.

Can Shopify Rollouts test prices or discounts?

No. Rollouts operates on published configurations — themes, checkout configurations and customer account configurations — so anything outside those, such as pricing or discount logic, is out of scope. It also applies to the online store only, not to headless or custom storefronts. Testing pricing or discount logic requires a server-side or custom implementation and the engineering capacity to maintain it.

The takeaway

Rollouts is a genuinely useful addition and a narrower one than the coverage suggests. It tests configurations, not elements, which makes it the right tool for redesigns, theme upgrades and checkout changes and the wrong one for a stream of small hypotheses inside a single theme. Take the free part regardless: gradual publishing to a slice of traffic is a release safety net that every store should be using, and it works from the Basic plan. Then be honest about the sample-size table before you buy anything on top. If a month of your traffic cannot distinguish a 10% lift from noise, the answer to “which testing tool” is neither, and the money belongs somewhere it can actually move the number.