Part 4: A/B testing

Testing is important to remove the guesswork and start making accurate decisions based on proof. If you're not running structured experiments on your ad setup, you're leaving optimization on the table and making decisions based on assumptions that may or may not be right.

Measure what matters

Measure the impact on the metrics that matter, like ARPDAU, retention and session length (learn more about this in our previous blog - Bringing your data together), taking it one variable at a time and running it long enough to trust the results. Really think about what you are testing and what the measure of success is for you and what the end goal is (increase retention, revenue, engagement - what exactly is it?). Most mediation platforms have A/B testing built in, and you can also run experiments through your remote configuration system if you want more control.

Splitting your users

Split your users into a control group (your current setup) and a variant group (the change you want to test). This is a crucial step and needs to be done with care and attention. Having biased AB test groups will heavily affect your results and lead you down the wrong decision path. Imagine doing a 50/50 split but one of your main revenue countries is included only in group A and not group B; or perhaps skewing one group by it containing all the whales while the others have none. If doing A/B testing through your own infrastructure take the time to analyse how users are split across groups and if it contains any biases.

Just as important is also knowing what kind of change you aim to be able to detect and plan size of your groups accordingly. For example, reliably (ie, with statistical significance, p < 0.05) detecting 3% change in ad revenue is quite a bigger challenge than detecting a 30% change.

When it gets complicated

Some tests are relatively contained like adjusting a timer or tweaking a reward amount. Others are a much bigger undertaking. For example, testing two mediation platforms against each other means handling SDK compatibility, creating and optimizing separate ad unit IDs for each group, and making sure your data pipeline can attribute performance cleanly to each side. Bidding and ILR (Impression Level Revenue) help with the data side of things, but the setup work is still substantial so you should know what you're getting into before you start.

What to watch out for

• Testing multiple variables at once is the most common mistake. If you change interstitial frequency and add transition animations in the same test you'll never know which one drove the result.

• Revenue metrics alone don't tell the full story. A 15% ARPDAU increase that comes with an 8% retention drop is not a straightforward win. Always track user behavior alongside revenue numbers.

• Group integrity matters too. If users are leaking between groups or seeing inconsistent configurations, your results aren't reliable. Make sure your testing system is actually keeping groups separated before you trust what comes out of it.

  
  

Write it all down

Keeping a proper log of what you tested, your hypothesis, the duration, sample sizes, results across both revenue and user metrics, and the final decision is more valuable than it might seem at the moment. User behavior shifts over time, so a test that flopped in Q1 might look completely different in Q4 with a different seasonal mix or a more established user base. Aggressive interstitial frequency that hurt retention early on might show neutral or positive results once you have a more engaged core audience. Without records you lose that context and you end up repeating work you've already done.