Wrangling the customer feedback overload
Faire’s four-step process for consolidating and prioritizing disparate feedback to systematically improve our product
Although product teams are often adept at outbound research like interviewing and surveying customers, they can find it challenging to fully leverage and analyze insights from inbound feedback such as support tickets and account manager conversations. In this article, we’ll share how Faire developed a system for quantifying and prioritizing such inbound feedback. We hope that other product organizations looking to get the most out of customer feedback and frontline team inputs will find parts of our approach to be useful complements to (not a replacement for) their traditional prioritization processes.
New customers, new opportunities to prioritize
After success in the US and Canada markets, Faire’s expansion into European markets in 2021 offered Faire brands the opportunity to connect with thousands of independent retailers across Europe. This expansion also introduced new and interesting challenges for our product teams. To ensure our French users had just as good of an experience as our American users, for example, we needed to address different preferences and needs across language, currency, payment methods, shipping carriers, support models, and more.
With so many areas to improve the experience for our customers, our product team had a prioritization challenge. Since many of these opportunities required significant investments — such as payment or shipping vendor integrations over multiple quarters, or overhaul of the UX flows — it was important to ensure we were prioritizing the right opportunities. We needed a way to collect, organize, and evaluate multiple sources of customer feedback versus just listening to whichever account manager was advocating the loudest for their customer’s specific need.
That’s how we ended up with our four-step process: (1) quantifiably collect the feedback, (2) categorize them into discrete root causes, (3) score each root cause on frequency, severity, and effort to fix, and (4) measure our progress.
Step 1: Collect inbound feedback in a format that can be quantified
At the time, Faire had a variety of customer feedback — such as support form submissions, account management conversation notes, customer commentary from experience ratings, and much more. The key was making all this information easier to quantify and analyze in aggregate. Here are some ways we did that:
Customer support requests
The idea: help your customers help you — and get to the root of the issue faster. For example, when contacting Customer Support, our customers would previously enter order IDs as free form text or even screenshots. Although our support agents could work off of this level of info, our product team couldn’t analyze what type of orders had a higher propensity for customer issues. Therefore, we updated our support request form, making it easier for customers to choose from any of their past orders, while also making the data available for our product teams to analyze.
Frontline feedback
Previously, our account managers and CX team members would share feedback with the product team by Slacking product managers directly. Although it was very collaborative, it wasn’t scaleable and didn’t allow us to objectively measure all the feedback we were collecting. Today, all of our frontline teams share feedback in a centralized tool which helps our product teams search for tags and keywords, as well as build alerts and dashboards that quantify topics they care about (we use a tool called Chattermill).
Step 2: Consolidate each piece of feedback into a discrete root cause
Speaking of tags and keywords, that’s the key second step: drilling down into the root cause of each issue and tagging it correctly. For example, if an order is delayed, it could be due to multiple causes: the brand (seller) being out of stock, the shipping carrier being slow, the package being stuck at customs, or something else entirely.
Getting to the discrete root cause of an issue requires truth-seeking rigor and collaboration, an important cultural value we practice at Faire. Given the complexity of our marketplace business, it’s best not to rely on metrics alone. In order to truly serve our community of customers, our product managers read support tickets to get a better understanding of who the customer is, what friction they felt, how it felt for them, and how to ultimately connect this information back to the product experience. Obviously, PMs can’t read every ticket — that’s why it’s important to build a relationship with your frontline teams and truly listen to them. They can advise your product teams on which tickets to read, can help curate tickets, and help identify common themes to translate a customer’s painful experience into a generalizable root cause.
Through this process, our teams have been able to distill tens of thousands of seemingly discrete pieces of feedback into a list of about 100 clear opportunities to address.
Step 3: Score each root cause to prioritize
While a list of around 100 opportunities can be prioritized more easily than thousands of free form feedback, it’s still too many to prioritize intuitively. That’s where our pain score framework came in, which helped us measure both the severity and the frequency of a root cause.
We pulled our frequency measurements from our quantifiable feedback (step 1). Also, for certain root causes (and symptoms), like packages getting stuck at borders, we could just count them. To be clear, “just counting” took operationalizing a special report with all our carriers, but we found these instrumentation investments to be worthwhile in honing our prioritization process.
Severity is harder to measure. We started with a subjective grading rubric and assigned scores to each root cause manually. More recently, our Data Science team’s causality model has helped us more precisely define severity as “how much a particular incident reduces the probability of the customer coming back to order again.”
Finally, we divide the pain score by the effort to solve the root cause (another subject rubric) to arrive at Pain Score ROI.
This equation helped us identify where we can make the most impact today. By sequencing our roadmap starting with the highest Pain Score ROI project and ending with the lowest, we have been able to improve our European customer satisfaction as quickly as possible with our available resources.
Step 4: Measure progress
Measuring progress is critical to ensuring your system is working. We measured progress holistically across three types of metrics:
- Feedback frequency and severity: Inbound feedback related to the particular root cause should decrease. This may be in the form of a reduction in frequency (fewer support contacts) or a reduction in severity (customers are less frustrated when they do contact). Again, we were able to monitor this because steps 1 and 2 ensured we could quantify these consolidated root causes.
- Product adoption: Solving a root cause usually meant improving the underlying product experience or even launching a new product. For example, when Faire improved the reliability of our system that generates shipping labels for our brands in Europe, we saw a dramatic increase in brands using our shipping labels rather than working directly with shipping carriers.
- Downstream business metrics: Although often hard to measure, the downstream business metrics should improve as well. For example, when we enabled our European brands to manage their catalogs in multiple currencies, we saw them optimize their listing prices more effectively — which subsequently helped them get more orders.
When these metrics didn’t move, we went back to tune our assumptions. We noticed that we estimated the frequency and severity well enough, but underestimated the effort required to solve. For example, we thought one solution would solve the pain point or issue, when the root cause, but instead a series of several complementary solutions were required — necessitating a bit of grit to fully address the underlying customer pain.
Measurable results — and more
Overall, in less than a year of operationalizing this new prioritization approach, we saw over 30% reduction in support contacts and a significant increase in customer satisfaction in blind surveys, where European customers rated Faire as the smoothest experience among all wholesale marketplaces.
In addition to customer benefits, we’ve also seen an increase in our support and account management teams’ satisfaction. This feedback process allows them to more clearly see the impact their work has in helping improve our customer experience, and enables them to focus on more complex customer challenges.
Given the positive impact we’ve already seen on both our customers and internal teams, our product team will continue to find ways to deepen our customer feedback channels, so we can ensure we’re prioritizing the tools and products that will help our customers succeed.
Curious about what it’s like to work for a world-class product team? Take a peek at open roles.
Thanks to Director of Business Operations Greg Rolfes, Head of Operations Shan Hu, and Design Manager, International Arthur Che for their help in co-authoring this piece.
