Product-Market Fit Survey Fail: How Experts Put Us Back On Track

P

Edit, 09/07/18: This story blew up a little bit! Special thanks to growthhackers.com, Tech Bound, and morph.ai for featuring it in their respective newsletters.




The product market fit survey is a pillar of the modern day startup.

Handed down to us mere mortals by the Holy Trinity (Mark Andreessen, Sean Ellis, and Steve Blank; for you pagans out there), the ‘PMF’ survey is the standardised test of startup viability.

About 6 months after releasing Jukebot, our collaborative Spotify playlist bot for Slack, my buddy and I decided to run the survey by our users.

This is the story of how we flopped. And how industry leaders helped us get back up.

Crash course: product market what?

Before we move on, let’s catch everyone up.

The product market fit concept is quite simple. It is a stage at which the solution you have built satisfies the demand of the market you’ve built it for.


This is one of Sean Ellis’ slides, onto which I’ve added the arrow.


A startup will go through many stages before reaching the holy PMF grail (I promise, I will stop with the religious comparisons in a minute).

Typically, one would

1) Notice a problem that needs fixing.
2) Build a minimum viable product (MVP) that fixes that problem.
3) Get feedback on that solution from people who have that problem.
4) Iterate based on said feedback.
5) Reach a problem/solution fit, where the solution built actually solves the problem.
6) Aim for product/market fit, where people will actually pay for that solution to solve their problem.

In very simple terms, a startup reaches product/market fit when people are willing to pay for the solution it is providing.

What is the product/market fit survey?

So, you’ve been through the five steps (and many more omitted ones) above and are ready to figure out if you’ve reached PMF.

One way to do this is simply by asking your audience. In today’s startup era, we don’t shy away from talking to our (potential) customers.

‘Build it and they will come’ is dead, long live ‘ask them what they want, build that, and they will already be there waiting for you to finish with cash in hands’.

The PMF survey can be grossly reduced as one simple question to your users:

How would you feel if we were to take this solution away from you?
– Very disappointed
– Somewhat disappointed
– Not disappointed
– N/A, I no longer use the solution

Sean Ellis, who popularised this survey, estimates at least 40% of your users should be very disappointed to see your solution go for you to consider having reached PMF.

Under 40%? You’re either not solving a real problem or delighting them enough; very unlikely people will pay.

Jukebot: what is it and why now?

Jukebot is a little side project my friend Joe built in his spare time. We thought it would be fun to build a virtual jukebox everyone on our Slack team can control.

It’s quite simple. You first integrate your Spotify. Then you integrate your Slack team. Then you create a collaborative playlist for your team, and you’re good to go. Everyone on your team can use Slack commands to add songs, remove songs, play music, turn volume up/down, and more.

> Grab Jukebot (it’s free)

We released it in the wild about 12 months ago, never expecting it would do much. We use it at the office every day. It is available via the Slack App Store, and was featured for a few days over Christmas last year.

Over that 12 months period, we grew entirely organically. Today, we’re at over 900 teams using it and hovering at 80 DAU (a user here being a Slack team, if we look at actual individual users we are at 5,238). Our website is rocking a cool 17.28% conversion rate, converting between 5 and 10 new teams every week day (weekends are typically very quiet).


Between these numbers, our fondness of the tool, and the tickets/feature requests we get every day; we thought there was an opportunity. At the back of my mind, Kevin Kelly’s ‘1,000 True Fans’ blog post also drove me to probe our users.

How would they feel about paying for the product? Where are we in the PMF charts, above or below the 40%? What insights can we gather from asking people’s thoughts?

We sent the survey via email to our entire list on May 17th. The results were… disappointing.

PMF survey results and immediate aftermath

We put together a 4-question survey:

  • The PMF question (see above)
  • How much would you be willing to pay for the service as is?
  • What feature do you think is missing from the service?
  • How much would you be willing to pay for the service should that feature be implemented?

We sent the survey to our list of 452 recipients and got…

*drumroll gif*

4 responses.

And, guess what? The almighty PMF question’s 4 answers were an exact 25% split.

alex debecker PMF survey fail


Crap.

Growth experts share their thoughts on our PMF survey failure

I was highly disappointed.

The low number of participants to the survey actually got us double-stuck. On one hand, the numbers were so low we couldn’t really gather any valuable insights from the survey. On the other hand, we had just reached out to everyone with an email blast and thus didn’t feel like we could bug them again any time soon.

Double-stuck.

So, I did the only thing I could think of: desperately reach out to industry leaders. Never in a million years did I think I would get 100% response rate from them. But I did.

morgan brown alex debecker PMF survey
Follow Morgan on Twitter @morganb


I reached out to them privately and they all answered. I also created a thread on GrowthHackers.com to ask a broader audience. Each time one industry leader answered to me privately, I urged them to also share their valuable thoughts publicly so everyone could learn. And they did.

> Read thoughts from Sean Ellis, Morgan Brown, Dani Hart, and Susan Su: Our PMF survey got 4 answers. What now?

What I’ve learned from running this PMF survey

Looking back, and thanks to the amazing feedback we received, we made two major mistakes.

1. Set high expectations

I was really silly. I honestly expected about 50 people to answer the survey. It’s a 4 question survey, how hard can it be to fill it out?

Turns out, conversion rates are really low for this. As Sean Ellis points out, we should expect a 10% conversion rate. We only got about 1%, which is way worse. Why? Queue to mistake number 2.

2. Unengaged list

Susan Su pointed it out and it was a real eye-opener. If your list is unengaged, your chances of getting responses go way low.

Surprise surprise, our list is absolutely not engaged. At all. We send one welcome email when people sign up and that’s it. Remember, Jukebot is a side hustle. We never really took the time to develop much rapport with the people who use the app.

Big mistake.

Action plan: making the best of the data

Ok, so we only have 4 survey results. Big whoop. Industry top dogs gave us a few ideas on how to pick ourselves up and turn this poor data into actionable insights.

Here are 4 ways to get useful data from our current users.

Personal outreach

When you want insights from your users, sometimes all you have to do is ask. I like this quote from Jake Knapp in his book Sprint:

jake knapp sprint interview
Page 200 of Jake’s amazing book, ‘Sprint’.


This is particularly true when the statistics you have are meaningless (‘cause of sample size).

So, action numero uno: reach out to our users personally. No more newsletter, these don’t work. Morgan Brown recommends picking a few users at various levels (power user, people who used once and ditched, etc.) and ask them questions.

This one is also echoed by Sean Ellis who recommends talking to the person who said (s)he would be very disappointed. It’s a starting point, use it!

Study the user data

If, like me, you have the chance to be working on a product developed by a genius of a dev, you could ask him to dig out the product users’ data.

What do people do with the app? That user ID who seems to be using the app every day, when did they get onboarded? What other information can we gather about their platform use? What can we learn from those users who joined and never created a playlist? What about those who added a single song then left?

It’s your app. Dig through the data. Find the power users yourself, understand the way they use the platform, see if you can find matching behaviours.

Fix stuff/reply to tickets

When we sent the survey, we got 4 answers. As the poor numbers started to (not) roll in, I decided to reach out to more experienced people and ask for their thoughts.

My buddy’s reaction? Fix stuff and email people back.

Again, this is a side hustle. We don’t always have time to fix bugs or add features, let alone email back & forth to enquire about users’ requests. But if we want our users to take us seriously, we need to start doing this.

Every reply he sent to a user, he added a personal request to complete the survey. We went from 4 answers to 8 in a few simple emails.

This shows the power of engaging with the users instead of shamelessly asking for something from them. If you are in this position, who could you email? Who has been emailing you about feature or bug XYZ? Now’s the time to get back to them.

Put feedback on automation

This is an easy one. I told you we never email our list. Now, we do. After two weeks, our users get an email asking for their thoughts on the app.

It’s a simple but effective one. Two weeks should be enough that engaged users will want to share some thoughts. It should also provide us with a constant stream of feedback from new cohorts of users.

Making lemonade

Jukebot is not at product/market fit. We didn’t expect it to be.

We wanted to learn from our users, see what we could do to turn our app amazing. It was disappointing to see such low engagement, but also very telling of the work we had to do to get there.

Thanks to the insights of great people, we’re back on our feet. We have a plan and a fantastic learning experience. Isn’t that what side hustles are really about?

Add comment

ALEX DEBECKER

Digital marketer. Company grower. Bearded internet warrior. I grow projects into profitable businesses.

Want to know more? Read this.

Recent Posts

FOLLOW ALONG

Social media is the best way to get in touch with me. Use the buttons below, follow, and please engage -- I answer every message.