The first step in the experimentation process is designing your experiment. So you start with your objectives and your goals. So let’s take an example, Suppose Instacart is trying to increase the retention rate and lifetime value of new customers.
So Instacart has a premium membership model where you pay something like $150 a year, and then you get unlimited free deliveries in two hours or less. If your objective is to increase lifetime value or retention rates of new Instacart customers, and you want to run an experiment around getting more people to buy into this premium membership model.
Your hypothesis would be, let’s say, increasing the number of emails about our premium membership model in your first 30 days after signing up with Instacart. That will increase the percentage of new customers who buy the premium membership model by ten because people will be more educated about the benefits of the premium model, in terms of price, convenience, etc.
So the essential components of any hypothesis are like the independent variable, the dependent variable, and your assumptions.the independent variable here is the number of emails or communications that you send to new customers about the premium product. So it could send two emails, three emails, whatever.
But what you’re trying to see here is, how does the independent variable, the number of emails you send, affect the dependent variable, which is the percentage of new customers who buyers who purchase the premium membership model. And then, you assume that people will buy more of the premium membership because they will be more educated about the benefits of that service to them, in terms of price, convenience, whatever.
The reason why assumptions are important. Because it helps people to think of their experiment through the lens of the customers.So rather than just sending them emails about the premium membership to get them to buy more stuff. Think, how will you send people additional communications about a premium membership to convince them, So that they will buy it.
How will you convince them is a valuable thing. So think about like, you want to educate them on like the value proposition in terms of convenience and you have to let them know that it’s going to save them money.It will help you to think about how will you going to experiment with designing the communications.
What do the users care about and then, how are we by delivering a better message and more clearly communicating this value proposition to our users going to improve this vital business metric and to achieve your objective that you want to have the hypothesis.
You could have some projected impact. You can improve it by 10% or 15%. Set up the experiment as an A/B test. You want to start by just testing, generally recommend to start, any time you’re experimenting on a certain like part of the funnel for the first time.
You start with the basic A/B test and don’t send them an email about the premium model versus send them an email about the premium model and see how much more of a lift do we get by just sending them the email versus no email.
Then you know what the value of an email itself is. After that, if you do see that there’s some lift, say 5% more people buy into the premium membership model. You start testing different copy, different email content, different design, whatever to then try to beat the existing version.
So like the next hypothesis could be talking about price is more effective than talking about convenience when it comes to the membership model. So test an email that talks only about how much money it’ll save, rather than how convenient it is.A/B check them against each other. Whichever one wins is the one you move forward with. And you keep repeating that process.
Even if you are wrong, then you can like to learn something valuable about your customers. So if you ran that test and you saw price doesn’t perform well, people care more about the convenience factor, then that’s great. You didn’t succeed with the experiment, but you learned something significant about your customers.
So you can use that to keep adding and building upon the process. That’s how you want to design your experiments. Step two is then implementing it. So ship the experiment, get it set up so you can measure the results, and then, step three is analyzed. So look at the key metrics of the experiment.
The most important one in this case for Instacart premium membership one is the percentage of treated users who buy the premium model. You could look at other things like email level performance. So did the email perform better in terms of like open rates and click rates? You can look at different metrics around like one metrics percentage of people that buy.
Still, you can also look at how many purchases did they make in a week, lots of other things that you learn about user behavior by looking at these experiments. And after you analyze it, if you find that it does work and that it is successful in the sense that you’re getting more people to buy the premium membership and you’re driving more revenue for the business, then you want to automate that.
So automation is an often overlooked, yet significant part of the growth process because everything you automate means that the value that you just delivered to the business, that incremental 5% in revenue, doesn’t need to get maintained by a person.
You don’t need to keep sending that email every week. You just permanently added 5% more value to the business. You automate it, you set it up, so it’s triggered based on the same time since sign up, you send them this email telling them to buy this thing, and then, you forget about it and try to come up with a better version of that.
But that’s how you start stacking wins on top of each other. And by automating it, it frees you up to then focus on what is the next experiment, what is the future growth unlocks that you can find.
So a lot of people think about email automation exclusively. There’s a lot of other exciting ways to leverage automation to improve your growth process and speed it up. Automating your analytics is a powerful thing you can do, and we’ll talk about it a little bit more when we get into optimizing your growth process.
But automation has become bigger and bigger, especially with like artificial intelligence entering the world of marketing. The more kind of mindless mechanical tasks that you could automate, the more time you can spend thinking about.
creative new ideas and better campaigns to run and then allow the machinery just to push out the emails, run the experiments in the background while you think about the next experiment to run.