Evaluating Gen AI's Product Market Fit
This is a multi-part research project I led at WhatsApp (Meta) to evaluate the launch of a new Gen AI chat feature and provide strategic insight for product iteration.
Skills Used:
-
Research Design
-
A/B Testing
-
Survey Design & Analysis
-
Diary Studies
-
Stakeholder Management
Some information has been removed or modified due to confidentiality.
Context & Research Goals
A new Gen AI Chats product was about to be launched across Meta's family of apps, including WhatsApp. This posed a challenge for WhatsApp, a brand built on simplicity and privacy to launch an AI product that didn't necessarily fit in with user's expectations for the app. I took on designing and implementing a research plan to give WhatsApp a comprehensive view of the user experience.
Methods
To ensure a thorough understanding, I proposed a three part plan:
1) A/B Testing to see how the new product impacted user's sentiment towards WhatsApp.
2) a User Experience Survey & Diary Study to understand breadth & depth of the experience and it's pain points.
3) a Product Market Fit Survey to identify the future use cases with the most potential to increase adoption & retention for WhatsApp users.
Crucial Insights
From this research, we were able to:
1) understand how the new product influenced WhatsApp user sentiment
2) identify what was & was not working with the product at launch
3) what use cases we should prioritize when iterating to acheive product-market fit.
Impact
In such a new product space, there was a lot of thrash across design, product, and engineering. By closely collaborating with these teams throughout the research process, I was able to land these insights were able to guide the direction this generative AI product is headed.
Key Learnings
Leaning into a fast, new space can be quite overwhelming at first, but if I just focus in on what I can get done now and show my value early, I can get stronger stakeholder buy-in and ensure users are heard.
Before designing the plan, I looked at previous launches and spoke to people working on the current AI product to learn what had worked well in previous launches.
This project was a top priority so employing a variety of methods at the right time allowed for a comprehensive understanding of the user experience.
The User Experience Survey allowed us to get a sense of the product experience at scale while the diary study allowed us to gain deep insights. Together, they gave stakeholders a tangible view.
Through our analysis, we found what use cases had the best fit for our users and drove product strategy to take full advantage of our user's appetites.
Stakeholders shifted frequently in the project, having centralized, approachable documentation allowed me to catch them up easily and make me a trusted partner.
For additional findings and learnings, please contact me at jackmizell12@gmail.com