Not-Scary Ways To Start Measuring Incrementality In Advertising

The last article defined what incrementality is and why it’s an important concept for advertisers to consider when planning and evaluating campaigns. In certain cases, digital ad platforms can overestimate their own value, taking credit for more conversions than the true incremental value of an ad campaign. Other advertising channels, especially non-digital ones, can be difficult to measure at all. 

There is no silver bullet that magically measures the exact value of ad campaigns. An advertiser needs to utilize a full gamut of tools to piece together a picture of what is working and what’s not in order to drive growth for a business. There are a lot of sophisticated tools and technologies out there that attempt to help measure advertising performance. You could get a PhD learning the methodologies for developing robust, scientific tests. 

However, as a practitioner running ads for a real business, you don’t always need absolute precision. There are a lot of ways you can run tests yourself, and as long as you’re doing the best job you can to limit how many variables change in the data samples you compare, you can get some good intel about your ad performance. This article will outline some of those tests you can use when you're unable to track a channel directly, or when you are suspicious of in-platform conversion metrics. I would start with some tests like these, and then depending on the results, invest in better measurement methods if you need more precision. 

Before/After Comparisons of New Channels

The easiest time to measure a channel’s impact across your marketing funnel is when you first start running the channel. As long as you start with a budget that is big enough to have a noticeable effect, you should be able to make a pretty clean comparison between your performance with and without the new channel.

For example, if you’ve never run retargeting ads before and you start a retargeting campaign on Facebook, you might see an overall bump in leads week over week. You might see the conversion rate of your search campaigns go up. You might see direct traffic or branded search tick up. All of these signals are indicators of the impact your new retargeting campaign is having.

Even though Facebook’s in-platform metrics for the retargeting campaign attribute 100 leads to that campaign in the first week, if you see that you are only up 50 leads week over week, you can guess that your retargeting campaign might be about 50% incremental (assuming the rest of your marketing activities have been pretty much the same week over week). If Facebook is reporting your cost per lead as $50, you can guess that your true incremental cost per lead from the campaign is closer to $100.   

Sometimes A Guess Is Good Enough

Any true data scientist or statistician would be quick to point out all of the flaws in this scenario. It isn’t a controlled experiment. But as long as you have a sense for the context of your business and marketing funnel, doing before & after comparisons can give you a good feel for the incrementality of a new channel. In the above example, your “true” incremental cost per lead is probably not $100.

If you did a more sophisticated experiment or invested further in measurement solutions, you might find your incremental cost per lead was $92 or $113. But given what you’ve been able to measure intuitively, your cost per lead likely falls somewhere in the $80 - $120 range. Your need to invest in more precise measurement depends on how important it is for your business to know where you fall within that range. If your leads on average are worth $500 in revenue to the business, it doesn’t matter whether those leads cost you $80 or $120. You should be making plenty of money either way. 

Heavy Up &  Holdout Tests

If you are trying to measure a channel that has been running for a while, you will have to engineer a test yourself so that you can get a read on the incremental impact of the channel. There are two broad ways to do this: heavy-up and holdout tests.

What is a heavy-up test in advertising?

A heavy-up test is when you increase the amount of spend in a channel dramatically in a short period of time in order to measure the incremental impact of the channel. Say you’ve been running TV ads regularly for the past 12 months, but haven’t been able to measure the impact of them very well. How much organic and direct traffic are they driving to the site? One way to find out would be to triple or quadruple your TV spend in a single month. All that extra spend isolated in one channel should create some amount of ripple effect across your other channels that you can measure. 

Say you go from spending $250K/month to $1M in the next month. You keep your other ad channels the same, and you notice during the test period that you got 1,000 more leads than normal, which would seem to suggest that the extra $750K of TV spend drove those leads at a $750 cost per lead. Again, all this is just an approximation, but it gives you a benchmark to allocate budgets better in the future.

What is a holdout test in advertising?

A holdout test is when you cut all ad spend in a given channel or geographic area in order to create a control group to measure against when calculating the incremental impact of advertising. A holdout test is the opposite of a heavy-up because instead of increasing spend, you cut it to zero. Once you have a period of time or an audience that you can confirm received no advertising from the channel, you can compare that to another sample where an audience was shown ads from the channel in question. The difference in conversions between your control group and test group is the incremental lift of the channel.

For example, say you question whether your branded search ads are driving any incremental leads for your business. So you turn them off for a week. During that week, you see a 10% dip in conversions. That would help you to start building a hypothesis for how much incremental value those branded search ads are driving.

But what if incrementality tests lower my ad performance?

Turning ads off or increasing spend quickly can be scary, and most likely it will hurt your overall efficiency for a short period of time. However, the return you get from running these tests is not based on whether or not your performance improves in the short term. The return is that going forward for the next 6-12 months, you will have a much better understanding of your ad channels and how to allocate your budget. Even if you lose $20K running a test right now, if that helps you allocate $5M 10% more efficiently for the next year, then it was all worth it. 

Just Get Started

Like I said at the beginning, there are tons of methods out there to measure advertising. But before you invest a lot of money into those solutions, run some tests on your own and see what happens. As you move your ad budget around, you’ll start to get an understanding of how your channels are working. Then when you invest in more sophisticated solutions you’ll already have a foundational understanding of your advertising program. 

One of the worst things you can do when advertising is keep your budgets the exact same in all your channels for an extended period of time. When you have no variation happening, it becomes nearly impossible to parse out the impact of each channel. 

Subscribe below and you’ll get the article in your inbox when it drops. And if you’d like to chat more about how to better evaluate the performance of your advertising, reach out with the contact form, and I’d be happy to have a chat about your situation.