Surveys are bad at initial learning

When asked to do the tiniest thing to learn from customers, many product people’s first instinct is to conduct surveys or focus groups. While running surveys and focus groups may seem more efficient than interviewing customers, starting there is usually a bad idea. 1. Surveys assume you know the right questions to ask. It is hard to script a survey that hits all the right questions to ask because you don’t yet know what those questions are. During a customer interview, you can ask for clarification and explore areas outside your initial understanding. Customer interviews are about exploring what you don’t know you don’t know. 2. Surveys assume you know the correct answers, too. In a survey, not only do you have to ask the right questions, but you also have to provide the customer with the right choice of answers. How many times have you answered [Other] to a survey question? The best initial learning comes from open-ended questions. 3. You can’t see the customer during a survey. Body language cues are as much an indicator of Problem/Solution Fit as the answers themselves. 4. Focus groups are just plain wrong. The problem with focus groups is that they quickly devolve to groupthink, which is wrong for most products. 5. Surveys can be pretty effective at verifying what you learn from customer interviews. A customer interview is a form of qualitative validation that is quite effective in uncovering strong signals for or against hypotheses using a reasonably small sample size. Once you have preliminary validation on your hypotheses, you can then use what you have learned to craft a survey and verify your findings quantitatively. The goal is no longer learning, but demonstrating scalability or statistical significance of the results. Inspired by the book: «Running Lean: Iterate from Plan A to a Plan That Works» Originally published at Startup Summary