Reading Price Experiment Results
Which metrics matter when reading a price experiment in Elevate A/B Testing, how orders are attributed, and the misreads that cost merchants money.
Price experiments are the easiest results to misread, and the most expensive to get wrong. This page covers which numbers matter, what they mean, and the mistakes that lead merchants to roll out a price that loses money.
Which metric decides the winner
The goal metric you chose at setup decides the winner. Every other metric is still calculated, so you can see the full picture.
Revenue per visitor
Almost always. Accounts for both how many people buy and how much they spend.
Profit per visitor
You have added Product Costs and want cost of goods, shipping, and fees subtracted. The most honest measure of a price change.
Average order value
You are testing something meant to change basket size rather than conversion
Subscription percentage of orders
You are testing subscription pricing and want to see the effect on subscription mix
The misread that costs money
A lower price almost always converts better. That is not a finding, it is arithmetic.
If you read a price test on conversion rate, the cheapest variation wins nearly every time, and you roll out a price that sells more units and earns less money.
A worked example. Two variations, 1,000 visitors each:
Conversion rate
3.0%
3.4%
Orders
30
34
Revenue
1,500
1,360
Revenue per visitor
1.50
1.36
The variation converts better and earns less. Conversion rate says the variation won. Revenue per visitor says the control did, and revenue per visitor is right.
This is why Elevate recommends Revenue per visitor for price experiments, and why Profit per visitor is better still when you have cost data.
What conversion rate means here
Conversion rate in your results is the share of unique visitors who purchased.
It is worth watching as a secondary number, because it tells you how buyers reacted to the price. It should not decide the winner of a price test.
Profit per visitor requires Product Costs
If Product Costs are not set up, profit columns and charts are hidden and replaced with an option to enable them. Add your cost data, and profit becomes available.
One caution: products without cost data contribute zero cost to the calculation, which inflates profit. If you enable profit reporting, make sure costs are complete for the products in your experiment.
How orders are attributed
A visitor is counted in an experiment once they have seen it. When they purchase, the order is credited to the variation they saw.
Assignment is sticky, so a visitor who sees a variation, leaves, and returns days later to buy is still credited to that variation.
This is why Elevate's numbers can differ from Shopify's or from Google Analytics. Elevate counts unique visitors and attributes revenue to the variation each one saw. Other tools count sessions, or attribute to the last channel. See Analytics Discrepancies.
When to trust the result
Elevate uses a Bayesian model and tells you when a result is trustworthy. Until it does, the numbers are provisional no matter how large the gap looks.
Three things that make a result look real when it is not:
Too little data. Early results swing hard. A twenty percent lift on forty orders is noise.
Too short a run. Weekday and weekend buyers differ. Run at least a full week.
A change mid experiment. If prices, targeting, or catalogue prices moved while the test ran, the data describes two different experiments. Start again.
Reading a result that is not a clear win
Not every price test produces a winner, and a flat result is still useful.
No difference at all usually means the price change was too small to matter to buyers, which is itself a finding: you have room to move without losing volume. Test a larger change.
Higher revenue, lower profit means the winning price sold products with worse margins. This is why Profit per visitor exists.
Higher conversion, lower revenue is the classic price cut result. You bought volume with margin.
Wildly different results by segment is worth investigating before acting. A price that works for returning customers and fails for new ones may be a personalization rather than a price change.
Before you act on a result
[ ] Elevate has identified a winner
[ ] The experiment ran at least a full business cycle
[ ] Nothing was changed mid run
[ ] You are reading Revenue per visitor or Profit per visitor, not conversion rate
[ ] Test orders you placed during QA are accounted for
[ ] The result matches or usefully contradicts the hypothesis you recorded
When you are ready, see Ending a Price Experiment.
Last updated