Split testing
Test one change at a time, tied to a goal. Don't ship on a hunch.
Last updated:
You watched the replays. You know the headline is too vague. You think a punchier one will convert better.
Think is the operative word. Before the new headline goes live for everyone, run it against the old one.
Pick exactly one thing to change
Open Split Testing and create a test. Three test types are available:
- Text for copy: headlines, button labels, microcopy.
- ID for branching code paths: control vs single-page checkout, sidebar vs no sidebar.
- JSON for structured config: a whole pricing object, a feature toggle plus a label.
Whichever you pick, change one thing. Not three. If you change the headline AND the button color AND the order of the form fields, and conversion goes up, you don’t know which one moved the metric. You moved three knobs and got one signal back. That’s noise.
Variant assignment is deterministic per visitor. The same person sees the same variation across sessions. You don’t need to think about it.
Tie the test to a goal
Conversions live in Goals, not on the test itself. Set traffic allocation (start at 100% if you want every visitor in the test) and weights (50/50 unless you have a reason).
When a visitor in variant A completes the goal, the goal fires and the variant gets credit. The dashboard shows assignments per variant and conversion rate per variant.
A test without a goal is a coin flip you can’t read. Create the goal before you start the test.
Wait for the Significance flag
The most common way these tests fail isn’t a bug. It’s the operator. Three days in, variant B is up 18%. You ship it. Two weeks later, variant A is up 4% and the lead is statistically real. But you can’t see that anymore because variant B is now production.
The dashboard shows numbers from the first visitor onward, so the discipline is on you. The number to watch is Chance to win: how likely it is this variant truly beats control. 50% is a coin flip. 95% is the threshold to ship. The Significance indicator at the top of the page flips to “Significant” the moment a variant clears that threshold.
The rule: don’t ship while Significance still says “Not yet.” Your founder-feeling that you already know the answer is the cheapest signal in the room.