Troubleshooting Product Experiments and FAQs
How to read product experiment results, fixes for common duplication problems in Elevate A/B Testing, and answers to frequent questions.
Reading the results
For what each metric means and how a winner is decided, see the reports and statistical significance documentation. Two things are specific to product experiments.
Check parity before accepting a loss. The most common reason a variation loses a product experiment is that it was missing reviews, an image, a variant, or an app, not that the idea was wrong. Confirm the two listings were comparable before drawing a conclusion.
A win is a listing, not an insight. If the variation changed the title and the description and the bundle, you have learned that the new listing sells better. You have not learned which part did the work.
The variation product shows no reviews
Likely cause. Duplicated products start with none. Reviews live with your review app, against the original product ID.
Fix. Use the message generated in the Product Reviews stage of the pre-launch review and send it to your review app's support team. Do this before launching.
The duplicate appears in search or collections
Likely cause. Its status was changed from Unlisted, or it was added to a collection.
Fix. Set the duplicate back to Unlisted in Shopify. Visitors should only reach it through the experiment.
The test product appears in my Facebook or Instagram catalogue
Likely cause. The duplicate is published to sales channels beyond the online store.
Fix. On the product page in Shopify, uncheck every channel except the online store.
Inventory or fulfilment is behaving oddly
Likely cause. The duplicate has its own inventory record and its own SKU, so systems that key on product identifiers treat it as a separate item.
Fix. Align SKUs and identifiers so all variations are recognised as the same product, and check that any app modifying product listings is applied to the duplicates.
The variation is missing images or options
Likely cause. Something did not carry across in duplication, or was changed afterwards.
Fix. Compare the two products side by side in Shopify and restore what is missing. Anything different that you did not intend is a second variable.
Results look wrong and the variation seems to underperform badly
Likely cause. Usually parity, not the hypothesis. Missing reviews are the most frequent single cause.
Fix. Work through the parity checks in Preview and QA a Product Experiment. If a gap existed for part of the run, the data before the fix is not comparable with the data after it.
FAQs
Does product testing need the theme extension? Yes.
Are duplicated products visible in my store? No. They are created Unlisted, so they do not appear in navigation, search, or collections. Visitors reach them through the experiment.
How many products can one experiment include? Up to 500.
Do reviews copy across automatically? No. Reviews belong to your review app and have to be transferred by its support team. Elevate A/B Testing generates the message to send them.
Can I test against a product I already have? Yes. Choose Select Product instead of Duplicate Product when pairing a group.
What is a group? Each product you select becomes a group, paired with the product it is being tested against. Groups keep the pairing explicit when you are testing many products at once.
Should I use product or product image testing? If you are only swapping photos, use a Product Image Testing Overview, which needs no duplication. Use product testing when the listing itself changes.
Which Shopify plans support it? All of them.
Next steps
Last updated