Create a Shipping Experiment
Step by step setup for a Shipping Experiment in Elevate A/B Testing, covering all eight creation steps, the Elevate carrier rate in Shopify, the preview sandbox, and launching.
This page walks through the eight steps of the New Shipping Experiment flow, then the four stage Experiment Review that follows it.
Before you start
Carrier-calculated rates
Your Shopify plan needs access to third-party carrier-calculated shipping rates. See Shipping Testing Overview.
Shopify shipping set up
The experiment works from your existing profiles, zones, and rates. Everything you select at Steps 2 and 3 comes from your own store.
Variations
Up to five, including the control.
A threshold plan
If you are testing a free shipping threshold and your site advertises that amount anywhere, plan for Step 7 before you start.
Step 1: General Info
Go to Experiments, select New Experiment, then choose the Shipping Experiment type.
Enter an Experiment name. This is the only required field. Name it for the comparison, such as "Free shipping threshold, lower vs current."
Add a Description and a Hypothesis if you want them. Both are optional.
Add Assets if useful. Images and documents up to 10 MB, videos up to 100 MB.
Under What do you want to measure?, select your experiment goal:
Revenue per visitor
The default recommendation, and the right choice for most shipping tests. It moves when conversion or order value moves, and a shipping change usually affects both.
Conversion rate
You care only about completing more orders.
Average order value
You are testing a threshold meant to push basket size up.
Add-to-cart rate
You are testing shipping messaging that appears before the cart.
Checkout start rate
You want to see whether shipping cost is stopping people at the cart.
Profit per visitor
You want margin rather than revenue. Only choose it if you have added Product Costs.
Select Browse all metrics for the full list, or to add a custom event. See Custom Events.
Remember that shipping fees count towards revenue on this experiment type. See Shipping Testing Overview.
Select Continue.
Step 2: Shipping Profile
Select the shipping zone the experiment runs in. Zones are grouped under the profile they belong to, and each shows how many countries it covers.
You select one zone. To test a second zone, run a second experiment.
Expanding a zone lists the countries inside it, which is the fastest way to confirm you are testing the right customers.
Edit zone opens that zone in Shopify.
Selectable zones must cover the same countries. Bypass country match lifts that restriction.
A warning sits at the top of this step and it is worth reading rather than dismissing: shipping experiments apply store-wide and cannot be limited to specific products or collections. Every product that ships through the zones you select will follow the test rates.
Select Continue.
Step 3: Shipping Rates
Select which of your existing rates the experiment replaces. Only rates from the zones you chose appear here, each shown with its current price or condition.
During the test, the rates you select are hidden and replaced with an Elevate A/B Testing rate adjusted for each group. Rates you do not select are untouched and keep showing normally.
Select Continue.
Step 4: Add Variants
Choose how many variations to run from Select the number of Variations. Five including the control is the maximum.
Under Set Variations Shipping Rates, each variation gets a card. Control holds your current rates and is not editable, which is what makes it a fair baseline. Each variation starts as a copy of those rates for you to change.
Use the pencil icon on a rate to edit it, the menu beside it for further actions, and Add another shipping rate to add one.
Rates group themselves by kind inside each card, so flat rates and conditional rates sit under separate headings. A conditional rate shows its condition on the card, which is the quickest way to confirm a threshold is set where you meant it.
The Add Shipping Rate fields
Shipping Rate Name
What the customer sees at checkout. This is customer-facing copy, so treat it as part of the test. "Free Shipping" and "Standard" read very differently.
Rate Description (Optional)
Supporting text beside the rate at checkout.
Shipping Price
The cost, in your store's currency. Leave it at zero for a free rate.
Delivery Time (optional)
Minimum and maximum day estimates.
Enable Conditional Shipping
Turn on to make the rate apply only within a range. This is how free shipping thresholds are built.
Conditional shipping works on two things:
Cart price
Applies the rate only within a range of order totals
A paid rate below your threshold, a free rate above it
Order weight
Applies the rate only within a range of total weights
Free shipping under a weight limit, a heavier rate above it
Weight conditions depend on your products having weights set in Shopify. A rate conditioned on weight will never fire on products with no weight recorded.
A Checkout Preview in the modal shows how the rate will read to a customer as you fill the fields in.
Keep rate names consistent across variations unless the name is the thing you are testing. If the control says "Standard" and the variation says "Standard Delivery," you have accidentally tested two things at once.
Select Continue.
Step 5: Traffic Allocation
Use the slider to set how much traffic each variation receives. The default is an even split and the percentages must total 100.
Isolate traffic for this experiment is optional and off by default. See Traffic Isolation.
Select Continue.
Step 6: Audiences
Narrow who is eligible, or leave the defaults to include everyone.
Device Type
All Devices, Desktop, Tablet, Mobile
Visitor Type
All Visitors, New Visitors Only, Returning Visitors Only
UTM Parameters
utm_source, utm_medium, utm_campaign, utm_content
Traffic
Source and Country
Create Custom Audience builds a reusable segment. See Audience Targeting.
Country targeting and zone selection are different controls that can contradict each other. The zone decides which customers get test rates at all. Country targeting narrows who enters the experiment. Setting a country here that your selected zone does not cover leaves the experiment with nobody to test.
Select Continue.
Step 7: Enhance Your Variations (Optional)
Skip this step unless your site displays shipping information that the test changes.
The common case is a free shipping threshold. If your site says free shipping above one amount and a variation charges differently, the customer meets a contradiction between your storefront and your checkout. Content Editing Changes lets you edit that messaging per variation so the two agree.
Content Editing Changes
Your storefront displays a threshold, a shipping price, or a shipping promise that differs per variation
Custom Code
You need JavaScript or CSS changes per variation
Click-Through Rate Tracking
You want to track clicks on specific buttons or links
Request Shipping Content Setup hands this to the Elevate A/B Testing team if you would rather not do it yourself.
Select Continue.
Step 8: Review Experiment
Check the name, goal, and every variation's rates. Then select Create Preview Experiment.
The experiment is created but not live. It moves to the Experiment Review screen.
Experiment Review
Four stages, ending in launch.
Overview
A summary of what you configured.
Elevate Carrier Rate
Elevate A/B Testing adds its carrier rate to the zones you selected when the experiment launches, so a zone already marked as enabled needs nothing from you.
To add it yourself instead:
Select Edit zone for the zone you want to change.
In Shopify, select Add rate for that zone.
Choose Use carrier or app to calculate rates, then select Elevate A/B Testing as the carrier.
Recheck zones refreshes the status after you make a change in Shopify. Once added, the rate appears in your Shopify shipping zone as Elevate Shipping Rate (Rates provided by app).
Preview
Two ways to check the experiment before it goes live, covered in full on Preview and QA Shipping Experiments:
Preview on a variation card puts you in that group so you can walk your own storefront as a visitor in it.
The Shipping Preview Sandbox lets you enter a cart total and cart weight, pick a variation, and see the shipping options and order summary that combination produces. It is the fastest way to check a threshold behaves at the boundary.
Elevate Review
The final stage explains what launching does: your current rates are briefly removed so the test rates can display, and your original rates are restored automatically when the test ends or is paused.
Then choose:
Request Review to have someone from the Elevate A/B Testing team check your setup before it goes live.
Launch Experiment to go live now.
Request Review is worth using on your first shipping experiment. Shipping is the one experiment type where a misconfiguration charges real customers the wrong amount.
Related
Shipping Testing Overview: requirements, how rates are delivered, and how shipping revenue is counted.
Preview and QA Shipping Experiments: the sandbox, the scenarios to test, and the pre-launch checklist.
Troubleshooting Shipping Experiments: rates missing, wrong rates, and messaging mismatches.
Hide Cart Item Properties: if experiment data shows in your cart or emails.
Ending an Experiment: ending a Shipping Experiment and what happens to your rates.
Last updated