brad dickason
PM at Facebook
home / about

Gather great feedback from your power users

On almost every product I’ve worked on, we’ve relied on a core group of active users to steer the development via great feedback and feature ideas. At Shapeways, most of our highly successful features came directly from our community. People who love your product are heavily invested in making it better. They will put in the time if you give them the space to do so and take their feedback seriously.

In this post I’ll share how teams have recruited, setup, and ran customer advisory councils (also called a customer advisory boards) that brought us a regular drip feed of high quality feedback. I’ll also share the mistakes we made along the way.

Key Takeaways

  1. To get the most relevant feedback, pick users who represent your target audience.
  2. PM should start the council and run it for a few months then hand it off once it’s generating high quality feedback.
  3. I recommend group / synchronous (e.g. video calls) sessions so you can ask follow-up questions and other users can chime in with their thoughts.
  4. You should actually listen to their feedback (and implement it). Otherwise they will stop participating.
  5. Bonus points if you can give them early access to features so they feel special (and you get early signal).

Why start an advisory council

As your userbase grows, it becomes very challenging to understand what people actually think about your product. You’ll quickly get to a point where you are relying on data and expensive user research studies just to keep a pulse on what job people are hiring your product for and where it’s falling over. The beauty of an advisory council is that you don’t just get to hear from some of your best users, they’ll gather feedback on your behalf from other community members. It is by far the most effective way to start to get real stories from real users that are passionate about improving your product to meet their needs. I’m not suggesting that it’s the only signal you should care about, but it should be a tool in your toolbox alongside data and formal user studies.

Here are some examples of how advisory counciles drove impact:

How to start an advisory council

Identifying your council

The first step is to identify who from your audience should join your council. You should start by defining who your target / mvp audience is and then filtering within that group. If you want to serve pro surfers from Australia, filter your userbase to that group and then look at their activity. You want users who are active, vocal, and constructive and represent your target audience. I typically will look at forums or customer service conversations to hand pick the first group. If your product has an interactive format (e.g. a game or a social media app), you should be able to spot folks using your product publicly. You want to avoid users who are constantly negative (they will drown out the voices of those who want to help) and have needs that are drastically different than one you plan to fulfill (e.g. if you are building a consumer 3d printing company for dog tags, you should include cat owners but not enterprise auto manufacturers).

Recruiting members

Reach out to each member and send them a personal e-mail or note. You need to make it clear that you’re handpicking them (so they feel special), that you value their input (so they feel wanted), and that you plan to involve them in the development process (so they feel like a part of the team). It’s important not to set expectations (e.g. we will ship your feature) because there’s a good chance you won’t use their feedback in the timeframe they expect. If you have a specific time commitment in mind or rewards you can offer (see 'How to get your council to love you' below), make them known upfront. I strongly recommend avoiding any NDA’s if possible. Yes, you’re letting them behind the curtain of the developent process, but you should be able to keep any extreme trade secrets out of the way and you WANT them to tell their friends about how awesome it was to be a part of this. This is a great way to grow via word of mouth.

Contacting members

Don’t use fancy tools like mailing lists or contact forms to communicate with your council. These make it feel very formal and like they’re just on another mailing list. You want them to feel like part of the team. Either stick to e-mail or use something simple like a Facebook group or Slack channel that facilitates group interaction. Make sure each email sounds personal and use an e-mail address that supports reply-to so they can send you open ended questions or feedback. You’d be surprised how many great ideas start as a random email reply to an event invite.

Who should run the council?

There are a ton of upfront decisions that need to be made to ensure that you get good feedback that is useful to the product team. Because of this, I strongly recommend that the PM sets up the council along with any early processes and facilitation. This will ensure that you personally get value from the group. Once the process is humming and you’re regularly getting great feedback, you can hand it off to I also suggest inviting at least your designer(s) and any eng that are product-minded. It’s more powerful to hear the ideas/feedback directly from the source than for you to proxy their comments.

How to run the advisory council

Ok, you’ve chosen your council and they’ve agreed to help. Now what do you actually do together?

Structure and cadence

There are a few options for how you run your council. You can choose between:

I’ve found that getting as close to ‘a group of people sitting in the room shooting the shit’ as possible is key.

I recommend a lightly structured synchronous group discussion over video. This allows you to facilitate in realtime and keep the vibe lighthearted (so it doesn’t feel like work) but still dig into meaty questions and hear multiple perspectives. To make this work, you’ll need to schedule time in advance and have a rough set of questions you want to cover as a starting point. You’ll also need to make sure that your team knows the ground rules: Can anyone ask questions? Are certain topics off limits? How do we talk about what’s in development right now?

Facilitating the discussion

When the discussion begins, people will look to you to lead it. Treat this like leading a meeting and assume that you need to drive the majority of the conversation:

  1. Start the meeting with a loose agenda (e.g. “We will share a few recent feature developments to get your thoughts, we’ll then open the floor for Q&A, we’ll leave 10 mins for you to ask questions”)
  2. Make it clear when you’re calling on specific people vs when you want anyone to speak
  3. Develop a protocol like ‘hand raising’ but online and try to enforce it. Many community members are going to be introverted and will get drowned out by the few talkative ones. Give everyone space.

The most important part here is the Q&A. You should come armed with a set of questions way too long for you to ever get through. They should typically have open ended answers (like most user research questions) and be very explicit. Try not to hand wave like “If we built a feature that kinda let you sorta do X, would you use it?” Instead focus on specific questions related to their workflow, their problems, and how they use your product.

One of the best questions I’ve asked is “What step of your workflow takes the longest and why?” (because it led to specific bottlenecks in our app that we could optimize) and one of the worst questions I’ve asked is “What is the one feature you want more than anything?” (because you will get a bunch of random not-thoughtful responses).

Michael Sippy, VP Product at Twitter, keeps his sessions laser focused on problems: “The point is that we’re supposed to be solving a problem for people, and understanding that problem on a really fundamental level,” says Sippey. “Only then can you create a product that actually resonates with the market, and a business that can scale beyond any one feature.”

Don’t be afraid to let the discussion wander, let multiple community members answer the same question, or let them discuss amongst themselves. One member will often be inspired by another and share really helpful insights, so make sure you encourage the group to interact.

If you're struggling with this test, I suggest reading the book 'The Mom Test' which gives a step by step guide to asking questions which will get at the pain point/problems customers are facing.

Handling ideas during downtime

When your council isn’t meeting, your community members will still have ideas and feedback. I’ve found that creating an asynchronous forum (e.g. Slack, Facebook group, etc.) can be useful for sporadic ideas so they don’t get lost. Try not to create a contract like “we will respond to your slack in 24h!” but your team does need to respond. If your users feel like they’re throwing their ideas into the ether, they won’t continue to do so. I typically will respond to the async forum once per day and then refer back to it when I am writing questions for the next get-together.

Asynchronous groups are great for getting quick answers to specific quetions. TalentBin used their council for small data-driven asks: “We wanted to ask, 'What are the average contract values you guys sell?' and get a quick response, so we chose 10 of the people who have come to all our dinners and who we felt closest to. It was easy to dash off and hear back immediately.”

How to get your council to love you

Here are some things I’ve tried over the year that led to very positive results (stack ranked by impact):

  1. Grant early access to features. This can be via feature flags, gatekeepers, or whatever system works for you. The important thing is that your council has a chance to try some features before everyone else and has a chance to give feedback (in a timeframe that you can act on it). Game companies are excellent at this - many MMORPG’s run entirely separate ‘Test Servers’ where users can access future content or balance patches for months before they go live. People will take the time to rebuild their characters on these servers and sometimes play there exclusively, just for the ability to try things out early on. (If you’re granting early access to features that can affect their production data, you should make this clear upfront, especially if your early features are bug-prone).
  2. Recognize them in your patch notes. Whether you fix a bug or add a feature that someone suggested, mention them (by username or first name) in your release notes so they receive recognition. You could even add a fun easter egg in-product if it’s not too much work like a small label with their name on it! Your council will love knowing that they affected the trajectory of your product (in the same way that people who fund city projects love to get a brick with their name on it).
  3. Run community events. My most successful council began with an in-person event where we flew the group out to meet in person (pre-COVID-19, obviously) and arranged an entire day where they felt like celebrities. They got to have breakfast and meet the team in the morning, hear talks from team members during the day, and try a very early version of future features in the afternoon. They went home with swag and everyone left feeling like they were a part of our product team. It was a blast and those users continued to give great feedback (and tell their friends) for quite some time.
  4. Be available. I was constantly surprised how sometimes our council members would light up when we responded to their ideas. Showing up and responding makes a huge difference. You’ll also find some odd cases such as one of our council members who enjoyed talking on the phone for an hour (and was clearly very lonely). Sometimes, you are more than just a PM collecting feature requests to your council.

"If anything, people should read the 'How to get your council to love you' section twice. The UX of someone who gives feedback is a critical flywheel. Make them feel heard. Tell them what they inspired, thank them when you push it live." -@michaelmuse

Here’s a great thread from an early eBay Voices member on why they participated despite no special treatment early on.

This sounds way too good to be true...

Q: What if we over-index on power users?

I do agree that over-indexing on power users can create a very niche product that can’t attract a new audience but I’ve rarely seen this be a problem at most companies. Typically, PM’s don’t talk to their users and teams constantly ship features they believe will help achieve their goals. IF you find yourself receiving way too many power user features, if might mean that they no longer resemble your target audience. You can always dial back the amount of features from the council that make it into your backlog or add more average users to round out the group and make it more universally relevant.

Q: Why doesn’t every company do this?

I’ve seen teams at big and small companies take this approach but there are many other ways to get high quality feedback. This just happens to be one that I’ve found effective time and time again. I suspect some people think it’s crazy to spend this much time talking to users and others believe the famous “If you ask people what they want, they’ll say a faster horse” quote. Personally, I’m running this same process with a small group of early adopters of my newsletter/coaching business :) Let me know if you want in the early council!

Q: What if we can’t ship enough features to keep them happy?

The reality is, if you are routinely fixing the top pain points of your power users, you don’t need to learn about the quantity/volume that you’re shipping. People often only want 1-2 things and if you can deliver that (or more), they’re incredibly happy.

Did you set an intention for last year? What was it? What else did you do that helped you spend more time doing work that you love? @bdickason

Get my newsletter. It features simple improvements you can make to improve your day-to-day PM life. From Product Vision/Strategy to Goals and Metrics to Roadmaps and everything in between.

Share

Post last updated: Jan 4, 2021

Posts