Popup A/B testing shows two or more versions of one popup to different visitors and compares how many sign up. A winner is real when each version has been shown often enough and one of them is very likely to be the best, not when it is ahead after the first afternoon. This guide covers what to test, how to read the results, and the question everyone asks first: what is a good popup conversion rate.
Why there is no useful average popup conversion rate
People search for an average popup conversion rate hoping for a number to aim at. We are not going to give one, because any single figure would mislead you. A popup's conversion rate is conversions divided by the number of times it was shown, and both halves depend on choices that differ from store to store:
- The trigger. A popup opened by a click is shown only to people who asked for it, so its rate is high by construction. A popup on page load is shown to everyone.
- The offer. Free delivery and a newsletter are not the same ask.
- The format. A game, a one-field form and a bar behave differently.
- The page and the device. A cart page on desktop and a blog post on a phone are different audiences.
- What counts as a conversion. A sign-up, or a click on a button you marked as the goal.
A published 'average' mixes all of these together. The only benchmark that helps is your own campaign last month, on the same page, with the same trigger. That is exactly what an A/B test gives you: two versions, same visitors, same days.
What to test first
Start where the difference is likely to be large, and change one thing at a time. With two changes you cannot tell which one made the difference.
- The offer. A percentage off against free delivery against early access. This usually matters most.
- The headline. Specific against clever: 'Get 10% off your first order' against 'Join the family'.
- The picture. A product photo, a lifestyle photo, or no picture.
- The button text. Say what the visitor gets.
- The number of fields. Email only against email and name.
Leave button colours for last. They are the famous test and rarely the one that moves the result.
Timing is a separate question. To compare triggers, read the What opened it panel in the campaign's Insights tab, which shows how often each trigger led to a sign-up; see when a popup should appear.
When a winner is real
Early results swing. With a few dozen showings, two sign-ups in a row can put a version far ahead, and the lead is gone by the end of the week. Three habits protect you:
- Decide the test before you look. One change, written down in a sentence.
- Run it through at least one full week, so weekdays and the weekend are both in the numbers. Seasonal weeks such as White Friday behave differently from normal ones, so do not carry a result from one into the other without checking.
- Wait for the evidence. In Tafsee Popup the results table shows each version's showings, conversions, rate, Lift vs control and Chance to win. The app's own advice is to wait for 95% confidence before picking a winner.
Chance to win is the plain-language version of the statistics: given what has been seen so far, how likely is this version to be the best one. A version at 60% is a coin with a slight lean. A version at 95% or more, after enough showings, is a result.
If your store has little traffic, be realistic. A test on a few visitors a day takes a long time to say anything. Test bold differences (two different offers) and skip fine ones (two shades of a button).
Auto-optimize: let the split follow the evidence
A fixed 50/50 split keeps sending half your visitors to the weaker version until you step in. With Auto-optimize switched on, Tafsee watches each variant's sign-up rate and, about once an hour, sends more visitors to the one most likely to win.
- No variant drops below 5% until there is enough evidence, so a slow starter still gets a chance.
- Showings each variant needs sets the minimum. No variant is judged before it has been shown that many times, and until half of it the split stays even.
- Call the winner on its own ends the test when every variant has enough showings and one is at least 95% likely to be best; it then gets all the traffic. Switched off, Tafsee tells you and you decide.
- What the optimizer did lists every look and its reason, and Look now checks at once instead of waiting for the hourly look.
While auto-optimize is on, it sets the split. Switch it off to set the shares yourself.
How to run a popup A/B test in Tafsee Popup
A/B testing is included from the Growth plan ($19 a month) upwards. On other plans the tab shows your original version and an upgrade card. See pricing.
- Open the campaign and go to the A/B test tab. Before a test it shows one row, Original, with all visitors.
- Click Add a version. It starts as a copy of the original.
- Name it, write one sentence in What are you changing?, and set Traffic to variant (%).
- Click Create test. The new version opens in the editor; change one thing.
- Publish. The test runs only while the campaign is live, and each visitor keeps seeing the same version.
- When the evidence is in, click Declare winner on the version you want to keep. It gets every visitor and the test is complete.
Two details that save confusion: the shares under Traffic split must total 100%, and a version you duplicate from the menu starts at 0% and is not shown until you give it a share. The full walkthrough is in the A/B testing guide.
After the test
- Write down what you learned, including the tests that showed no difference. 'Free delivery beat 10% off on the cart page' is worth more than the winning popup itself.
- Check the leads, not only the rate. A version that collects more sign-ups with a bigger discount may not be the better business. Compare what those leads go on to do in your store or email tool.
- Test the next idea. Good popups are usually the result of several small tests, not one clever one.
No plan with A/B testing yet? The Insights tab has a What to try next list, which only suggests changes your own data supports. Read more in results and analytics, and see popup examples and ideas or lead generation for things worth testing.
Frequently asked questions
There is no single honest number. The rate depends on the trigger, the offer, the format, the page and the device, and a click-opened popup will always look better than one shown to everyone. The useful comparison is your own campaign against its previous version on the same page, which is what an A/B test measures.
At least one full week, and until one version is at least 95% likely to be the best after every version has had enough showings. On a low-traffic store that can take several weeks, so test large differences such as two different offers.
The Growth plan at $19 a month and the plans above it. Paid plans have a 14-day trial. On the Free and Starter plans the A/B test tab shows your original version and an upgrade card.
It moves the traffic split for you. About once an hour Tafsee sends more visitors to the variant most likely to win, keeps every variant at 5% or more until there is enough evidence, and can end the test by itself when one variant is at least 95% likely to be best.
