Jump 450’s Post

Transcript

So the easy way to think of it is an incrementality test is set up very much like a pharmaceutical study. So when there's a control and exposed group, right, we'll cover the placebo. So in those cases, some patients get the actual medicine, some patients get a sugar pill, right? And then they measure the impact that the pill had. No, naturally when you do those kind of studies, the goal is, well, the medicine should work quite well, right? Then there's those cases where like people who think they got the medicine right still show a lift. That's why it's called the placebo effect. Yeah. But because there might be a lift like that, you need to do a control and exposed experiment. So we do the same thing on the marketing side. Now the easy way to do that yourself is to create your own control and expose groups. And the easiest way to do that is through Geo test. So that if you want to think of the US for a second, there's 50 states, right? An easy thing to do, split the country in half. It's not as arbitrary as that. I'm just a very simplistic example. In a perfect world, you are literally going down to the DNA level. So like the the different metro areas, right? So like the New York DMA, Pittsburgh, Philly, all that. And then you're actually using your own data with more statistical modeling on top to then try and pick out these regions because the goals you want these regions to be similar. So that's why you can't just flip the country. Now, if you can't do like West side of the country and he signed the country, that's not good. You also can't pick North and South of the country. Right, there's gonna be different behaviors. And how states interact with things, right? Be the political divide in the country, right, and blue. So that's another thing hesitate take into account, like your products may skew one way or the other, right? So because of that, if you were doing it at a state level, what I try and do is try and pick two states that you know are similar, that may have similar population sizes as well as somewhat similar demographics. Like California, New York, while they are in two different coasts, are fairly similar, right? They're fairly similar in terms of their blue political leaning, they're fairly similar. In terms of they've got some rural areas, but they also have some big metro areas, right? So you try and find things that are similar to each other to put them into these buckets because you don't want all of your biggest states in one bucket and all the small states in another. Because you also need to make sure your control has enough weight in it to compare against, right? You need a sample in both. You can't just have like 1% of your data in the in the control and then the other 99% exposed because you won't, you won't feel confident enough in the control. So you need a big sample, right? So because of that, you want to. Similar so California and New York. Maybe Florida? Texas. Right, but then you might have to do something like. I don't know, Illinois versus North Carolina or something, you know, like what you're starting to get smaller, then you eventually have like Vermont versus New Hampshire, right? You're going just keep going down the list. And that's like a very rudimentary way to do it. In a better world, you could still do it by state, but you'd pull your data by state, like all of your sales by state and then try and break them out that way, right? Because even though California, New York might be their biggest states in the in the country, they may not be your two biggest state states in terms of sales. So you can, you can use your own data as well, right? But the whole point is you have to figure out what's my control going to be and what's my exposed going to be. Now they don't have to be equal. But you want to at least have like 20% or more in that holdout. So if this is states, you'd probably want at least 10 or 15 states in the control with a mix of blue and red states, the mix of high population, low population holds, Chiang kind of thing. So at least 15. I aim for 20. So I would like 30 in my exposed and 20 in my control. Sometimes you do 3515 something like that, but you wouldn't do like 45 five, right? So that's the first step, picking your reach. Because now what's gonna happen is you're going to send ads to one of those regions, the exposed regions, and they're going to send no adds to the control regions. Now when you do that, we'll then be able to measure with your own internal data, not the ad platform data anymore because the ad platforms are going to say you got no activity in the controls, but in your, in your own back end of your site. You know how many purchases you got, You know where they're coming from because you know, then Matt ordered something from California. You know that Tom ordered something from New Jersey. So you can then measure the pre during post. Three different numbers of how the sales were for those regions before I started running the ends, while I was running the ends and also after too, because you're going to have some lag and some Halo after time goes on. So once you have all that data pulled to see the pre during post at the control of exposed, you'll be able to figure out what's my lift. Because if you had 100 sales beforehand for control region and then in the timing of the campaign, your control region had let's say 105 sales. But then you're exposed region had 210 sales during that time. Well, look at the lift. It did doubled, right? Yet 105 purchases in the control. So that's almost like what would have happened in my regions had I have not run in the ants. I'm simplifying this. It's not exactly the numbers, but like it's probably around the 105 number. OK. But now my real number was 210. So the list is that difference, right? So the goal is, can you have a list? You may see a lift immediately depending on how impulsive your product. Like a shorter product cycle means if you run an ad today you may see the results quicker, but a longer cycle, you may need to run that test for a couple of weeks or months and then even measure after that because you may not see as big of a lift during the campaign. But you may see the lift come after the campaign for various reasons. Like imagine here's an example. Let's say you're in a campaign for two weeks. But you heavied up for two weeks. OK, let's make it a very specific example. Say the Super Bowl. A lot of people won't answer the Super Bowl, right? Let's hear the Super Bowl goes on February 3rd, right? Because it's using the first week of February. And let's say I'm a CPG brand that is just selling. Standing Pepsi used it before, right? Let's say you're selling Pepsi or anything you buy on the shelves. If I spend a bunch of money for two weeks up until the Super Bowl. I may not see those sales for Pepsi until the 15th of the month before the 30th of the month in February now. Why do you think they like? Because of awareness or people are people are sort of seeing the ads and then thinking about the thinking about the purchase before they. But would that be true of Pepsi? Yeah, it can be, but it's actually, it's almost Dumber than that if you when you think about it, which is when you're buying stuff in a store. You're not going to the store every minute of the day, whereas what our computers every day at all times. So I can easily go on an ecommerce site and buy something, a Pepsi, I don't buy all the time, right? So if the Super Bowl's on the 3rd and I've been running a bunch of ads for the past two weeks. A lot of people go to the supermarket after they get paid. Which is on the 15th of the month and some people only get paid once a month and they might get paid at the end of the month. So you may have to track for the next 4 weeks. Because those ads may have impacted them. Like Oh my God I love Pepsi but I'm not going to the store yet. So then they finally buy 28 days later and you need to make sure you measure that too, right? So that's why it's important to do a post.