Ep 9: Testing a new demand partner properly

You added a bidder and revenue rose. That’s not proof yet.

A holdout is a random slice of traffic kept without the change, for comparison.

On recipes.example, you switch on a new SSP on Monday, and revenue rises five percent. But the holiday season just started, and a recipe went viral. You can’t separate the bidder from the calendar. Do it properly. Pick visitors at random. Twenty percent get the new bidder, eighty percent are the holdout. Both groups live through the same days. After at least two full weeks, compare site-wide revenue per thousand sessions, not the bidder’s dashboard. Say the test group earns ten dollars thirty, the holdout ten dollars. That three percent is the real lift. Check page speed too, and after rollout, keep a small holdout.

Before the next partner goes live, write down the split, the dates and one metric.

Next, incremental versus cannibalised revenue.