In the past, I've worked with teams where the response to every growth idea is “let’s test it!!” - and we immediately start an AB test.
While this fast-testing mentality sounds cool, that fact is, most of those experiments are meaningless.
Without a proper growth experiment process, we could easily start experiments, but we weren’t able to actually complete them. So “experiments” would run indefinitely, or need to be repeated several months later when someone else has a similar idea.
Over the last few months at Sendspark, we’ve nailed a growth experiment framework to quickly test and track growth ideas.
We’re still able to get experiments up and running quickly, but we have a lot more organization and clarity around them.
With our growth experiment framework, we’re able to...
- Easily prioritize growth experiments
- Complete experiments
- Track results from past experiments
This framework originally came from 500 Startups Entrepreneurship in Residence, Timur Daudpota. We’ve made a few modifications for our business: Sendspark is a product-led B2B software to help sales and marketing teams increase email engagement with personalized videos for customers.
Our Growth Experiment Framework
Copy our growth experiments tracking spreadsheet here!
Then, either watch this video or read through our process below.
Our Growth Experiment Process 👩🔬
1. Prioritize growth ideas
Whenever we have growth ideas, we add them to our Growth Experiments Tracking sheet.
These are the fields we fill out:
- Experiment Title: a brief way to refer to the idea - ex: “Exit Intent Modal”
- Description: a little more context, if necessary - ex: “when someone moves their cursor to close out of the Sendspark website, we can show a popup that asks for their email address”
- Hypothesis: the benefit we believe this will have for our business, always in an if/then format - ex: “if we implement an Exit Intent Modal, then we will increase our website conversion rate”
- Owner: the name of the person responsible for running this experiment (remember - if you have multiple “owners” then you do not have an owner!)
- Category: which stage of the funnel does this experiment target?
And then we give each of them an ICE score. The ICE score is a score 1 (low) - 10 (high) for the three categories:
- Impact: If this experiment is successful, how big of an impact will this have on growth? 1 might be for an edge case, while a 10 would be for something that sparks a viral loop.
- Confidence: How confident are we that this experiment will be successful? A 1 might mean that there are factors outside of our control, like we’re relying on a 3rd party, and a 10 might mean we’ve already had a lot of customer requests for this, and we just need to release it.
- Ease: How much time or money do we need to spend on this? A 1 might be a big development project (mobile app, platform integration) while a 10 is a quick website A/B test I can start running in 15 minutes.
The three components are added together to come up with a total ICE score. We prioritize growth experiments based on their ICE score, and tackle the low hanging fruit first.
2. Start growth experiments
When we start running an experiment, we...
- Update the Status to “👩🔬 Testing Now.”
- Add a Date Started date
- Add a link to an Amplitude chart to track the KPIs as they come in
3. Complete growth experiments
Once the results are in, usually after 2-3 weeks, we complete experiments by doing the following:
- Choose a winner for the A/B test
- Update the Result column with “Success,” “Failure” or “Inconclusive”
- Write more context that we’ll want to remember if we ever consider running a similar test again. For example, “Success! With this bounce modal, we increased our website session → contact conversion rate from ~2% to closer to ~4%.”
A few more recommendations
Timing
Right now, our growth experiments are very tied to the developer sprints. Even though not all of the growth experiments require development, we find that this adds a little more consistency to the process. It also makes it easy when we do need to add something to the development sprint.
Tools
If you’re looking for tools to actually run and measure AB tests, I’ll make some recommendations here. You can also refer to this recent blog post for 23 SaaS tools to grow your startup.
To run A/B tests on your website:
- We use Hubspot (and have our website built on Hubspot)
- I’ve also used VWO in the past - it's very good and easy to use
- I haven’t used Google Optimize - it’s probably a little less good, but it's free.
To measure KPIs:
- Segment is a must for tracking use events. Once you track that data (“User Signed Up”, “User Upgraded Account”), you can send it any of the other SaaS platforms your use.
- We use Amplitude right now - it’s great and has an amazing startup plan (free for 1 million events)
- I’ve used Mixpanel in the past. It’s also good, but more expensive.
- I also tried using Google BigQuery and Google Visual Studio, but it was too much work. Definitely start with Amplitude, Mixpanel or something similar unless you’re a real data scientist.
Now run some growth experiments!
Let me know how it goes! If you have any recommendations on ways we can improve our process, we’d love to hear them.
