Troubleshooting Checkout Experiments
Fixes for Checkout Experiments in Elevate A/B Testing when components do not appear at checkout, one component renders and another does not, visitors are not recorded, or results look wrong.
Symptom first fixes for Checkout Experiments. Each entry gives the likely cause, how to confirm it, and what to do.
Most checkout problems come from one place: a component's extension has not been enabled in the Shopify Checkout Editor. Start there before investigating anything else.
No components appear at checkout for anyone
Likely cause: the component's extension is not enabled in the Shopify Checkout Editor. An experiment can be created and launched without it, and nothing will render.
How to confirm: open the Shopify Checkout Editor and look for the Elevate A/B Testing extensions in the extensions list. If the extension for the component you configured is switched off, that is the problem.
Fix: enable the extension, save, then start a fresh checkout in a private window and check again. See Create a Checkout Experiment for the full sequence.
One component appears but another does not
Likely cause: extensions are enabled per component, not per experiment. If you tested Checkout Banner last month and added Customer Testimonials this month, the banner renders and the testimonials do not, because that second extension has never been enabled on this store.
How to confirm: in the Checkout Editor, check each component's extension separately rather than assuming one switch covers them all.
Fix: enable the missing component's extension and save. Once enabled, it stays enabled for future experiments using that component.
The Checkout Experiment type is unavailable
Likely cause: the store is not on Shopify Plus. Shopify only permits checkout customization on Plus, so this is a platform restriction with no workaround.
Fix: on a non-Plus store, test the steps before checkout instead. Cart and product page changes are reachable through Page, Content, and Custom Code experiments.
I go to checkout and see nothing, so I assume it is broken
Likely cause: you were assigned to the control, and your control has nothing configured. That is the experiment working correctly.
How to confirm: open several new private windows and start checkout in each. With an even split you should land on a configured variation within a few attempts.
Fix: none needed. If you never see a configured variation after many attempts, treat it as the first symptom on this page.
I keep landing on the same variation
Likely cause: variation assignment persists for a visitor so the experience stays consistent. Reloading will not reassign you.
Fix: use a new private or incognito window for each check, or clear cookies for your store between attempts.
Components render but the report shows no visitors
Likely cause: tracking is not reaching Elevate A/B Testing. The components are delivered by the checkout extension, but visitors and orders are recorded by the Web Pixel, and those are two separate mechanisms. One can work while the other does not.
How to confirm: check that your installation is healthy on Verify Your Installation.
Fix: repair the installation, then confirm a new test order appears in the experiment's results.
The experiment will not launch
Likely cause: the extension step was not completed. After Create Preview Experiment the experiment waits on the Experiment Review screen until you enable the extension in Shopify and select I Enabled the Checkout extension.
Fix: return to the Experiment Review screen and complete both actions, then select Launch Experiment.
The component looks wrong on mobile
Likely cause: layout settings that work on a wide screen reflow on a narrow one. Horizontal badge rows and horizontal upsell layouts are the usual culprits.
Fix: check every variation on a phone as well as a desktop, and switch the affected component to a vertical layout if the horizontal one crowds. The preview panel in the app shows the component's appearance but not how it sits on a real narrow screen, so verify on an actual device.
The upsell shows nothing, or shows products I would not have picked
Likely cause: Auto Recommended Products hands product selection to Shopify's recommendation system, which draws on the cart contents and Shopify's own data. On some carts it returns little or nothing useful.
Fix: switch that variation to Offer Specific Products and choose the products yourself. That also makes the test repeatable, since the offer no longer changes from cart to cart.
The numbers do not match Shopify
Likely cause: for Checkout Experiments, conversion rate is completed orders divided by checkout starts, while Shopify divides orders by sessions. Both are correct and they will never match.
Fix: none needed. Compare your variations against each other in Elevate A/B Testing, and use Shopify's figure to judge your store overall. See Reports Overview.
Results are barely moving
Likely cause: only visitors who reach checkout enter the experiment, so the audience is a fraction of your traffic. Audience targeting narrows it further.
Fix: run on all visitors rather than a segment, and test a change large enough to produce a detectable difference. Expect checkout experiments to take longer to reach significance than storefront experiments. See Statistical Significance.
FAQs
Do I need Shopify Plus?
Yes. Checkout customization is a Shopify Plus capability, so Checkout Experiments are unavailable on other plans.
Do I need the theme extension?
No. Checkout components render through Shopify's checkout extensibility rather than through your theme.
Can I preview the experiment before launching it?
Not on your live checkout. The preview panel in the configuration step renders each component as it will appear, but there is no way to view the experiment on your real checkout until it is running. Verify immediately after launch instead.
How many variations can I run?
Up to five, including the control.
Can I configure the control?
Yes. Leave it empty to test a component against a checkout with nothing added, or configure it to test one version of a component against another.
How many badges, testimonials, or upsell products can I add?
Up to three of each. Payment icons are not capped at three, so list as many payment methods as your customers use.
Can I test more than one component at once?
Yes, and on low checkout traffic it is often the practical choice. The trade off is attribution: a combination test tells you the combination won, not which component earned it.
Can I run a Checkout Experiment while other experiments are running?
Yes. If you need certainty that another experiment is not influencing this one, turn on traffic isolation when you create it. See Traffic Isolation.
Which goal should I choose?
Revenue per visitor suits most checkout tests. Use average order value for upsells. Profit per visitor is selectable at any time, but only use it if you have added Product Costs, since without cost data the profit figure will not reflect your margin.
Related
Create a Checkout Experiment: the seven step setup walkthrough and the extension steps.
Checkout Testing Overview: requirements, components, and what this type cannot do.
Preview and QA Checkout Experiments: verifying variations after launch.
Ending an Experiment: ending a Checkout Experiment and implementing a winner.
Last updated