FUNDAMENTALS OF PRODUCT MANAGEMENT: THE PROCESSES
A framework for testing ideas during the product discovery process
Any worthy product discovery idea needs to be tested for 1) value creation, 2) usability, 3) feasibility, and 4) viability with the product’s various range of internal and external stakeholders. This is an arduous task, but a must for any product’s success
For success in product discovery, it’s crucial that product teams quickly separate good ideas from bad ones. By definition, good ideas are those that have competent answers to the following questions, in the order of priority below, which are fundamentally the risks we want to mitigate:
- Value creation: will the customers choose to use/buy this product?
- Usability: will the users figure out how to use this product?
- Feasibility: can the engineering team build this product?
- Economic viability: will this product bring profits to our business?
- Ethics: will the outcome be a net positive creation for society?
1. Testing for value creation
Customers will only purchase or use your product if they perceive gaining value. Therefore, just because someone can use your product (i.e., passes the usability risk), there is no guarantee that they will choose to pay for it or remain loyal.
This becomes a hard sell when a service or product wants customers or users to switch from other products or systems they are currently using (i.e., switching costs) — and it’s most often the case that customers will need to switch from a current product to some other.
Therefore, the value that your product creates needs to convince people economically, socially, psychologically, and emotionally to switch and remain loyal
Therefore, because of the complexity of value creation, just matching the features of current players in the marketplace (i.e., achieving feature parity) or lower prices generally don’t sell a product. Your potential target users or customers must perceive your product to be substantially better than what they currently have to get motivated into buying it and becoming loyal in addition to the pains of migrating from current subscriptions or solutions.
In action, successful and effective product teams should spend most of their time creating value. If a product creates value, most other challenges will be overcome including usability, feasibility, reliability, performance, and viability. Many examples of these are found in recent high-growth products such as Facebook or Uber that have overcome substantial legal and viability risks to IPO because fundamentally, they created immense economic (i.e., measurable) value for customers.
Quick test for meaningful demand: testing for the “what”?
When testing for demand, we need to conclude that the problem we are solving is something that people care about and there are enough numbers of them for it to make economic sense. Doing this check is usually an easy process and can be done at a feature all-new product or business level.
- For features, we can test demand through a fake-door experiment whereby a desired feature that we think customers might like or that we’ve benchmarked off the competition is placed within the product. When users click, it mentions that this is a study/experiment. By collecting user data and analytics through their clicks, we can see the conversion and click-through rates and conclude demand by comparing the fake-door’s performance to that of other features on our product. For this to work, users must not understand that the feature is fake until they have clicked on the button or feature.
- With new product ideas, we can use the fake-door concept but rather than build a button or minor feature, we need to set up landing pages that feel like a complete product with a specific value chain that is solving a problem. We need to set up the experiment to collect performance data which needs to be carefully thought about and may include surveys, email sign-ups, idea or interview requests, etc.
These tests provide great evidence regarding customer demand and potential leads for future contact. However, the problem with these experiments is that while people may click or sign up, the evidence is still not sufficient to conclude that they will be willing to change their current habits and pay for your product.
Testing for quality: testing for the “why”?
While testing for demand can tell what is and isn’t happening, it doesn’t tell us why it is happening. Meaning, that if customers are not responding the way you had hoped towards a feature or product, then you need to figure out why that is — and this is termed qualitative testing or testing for quality.
Testing for quality is not about proving anything, but listening to customers and rapidly learning and uncovering big insights, most often, this is the most revealing and insightful journey within product discovery for product managers and should be performed as often as 2–3 times per week. A qualitative test could look as follows:
- An initial screening interview to validate that the customer/user experiences the problem you are solving and is important for them
- Have the full team for shared learning. The team needs to have product managers, designers, and engineers to maximize the learning
- Conduct a usability test, to make sure that the customer/user understands your products or high-fidelity prototypes, can figure out how to use them, and gets acquainted with them if not already. A value test without a usability test will become a hypothetical and imaginary conversation that lacks the rigor of going deep into what features, how, and why are creating value
- Test for economic value or money. When the customer/user has experimented with your product/prototype, you will need to gauge their willingness to pay to see the perceived economic worth of your solution/service to that specific customer/user
- Test to social value or reputation. Customers/users should be willing to recommend a solution to their network of acquaintances, friends, family, and peers through social media for example. If not, then it’s not creating meaningful value for them
- Test for time value or excitement. With products of high value, customers/users should be more than happy to spend time testing or helping product teams bring their solutions to market. Otherwise, the value is most likely not there, at least as much as you would have hoped for
The goal of this process is to uncover as many meaningful insights as possible rapidly iterate the product/prototype and take it to more customers/users for further qualitative feedback.
Testing for significance in demand
When there is a feel for enough scale in demand and we are happy with the insights regarding why customers/users desire our product, it’s time to collect statistically significant evidence of this demand to make an informed investment decision.
This evidence can manifest itself in the form of traffic, engagement, and payments. While this might be easier to gather in established product settings where customers/users are more readily available, in startup settings, due to lack of scale in demand, the product team needs to take more intuitive risks and find better methods of measuring significance in demand while also convincing investors.
The two best tools to test for significance are Discovery A/B Testing and Invitation-Only Tests:
- Discovery A/B Testing: this is a great testing technique because users don’t know which version of the product they are seeing/using and therefore yields rather accurate and predictive results. What generally happens in discovery-related A/B tests is that we show the current product to 99% of the users (depending on the scale of traffic you have the share could change) and a live prototype or product to-be to the 1% of users or less for analytical feedback. There is also optimization A/B testing that is generally about measuring the performance of various CTAs, color schema, and so forth and it’s normal to have a 50:50 split between the test group and the actual product users as the risk levels are quite low for the business
- Invite-Only Tests: if the risks of testing are higher and you don’t have enough traffic to test yet, then invite-only discovery tests could be the option. This is where you identify a set of customers/users and invite them to try out the new version or product in private and experimental mode. The one note to consider is that the data collected from Invite-Only Tests are not as predictive as blind A/B tests and the customers/users are generally early adopters and innovators and inherently that could lead to bias later on as the early majority for example are not as enthusiastic as you would have thought. An option here would be to reach out to members of the customer discovery program and gain their responses to your new product
Getting on top of analytics for value analysis
It’s important that product managers clearly understand the role of data and analytics and with the help of various tools at their disposal, bring insights to the table to prove the value their products are creating.
Data analytics can be used to:
- Better comprehension of customer/user behavior
- Measure the progress products have made on objectives
- Prove whether features and ideas stick and work
- Inform product decisions and move teams away from opinions
- Inspire new areas of product work
2. Testing for usability
With every progress, testing for usability becomes a more mature and straightforward process for product teams. The only note is that usability testing needs to happen during product discovery and not after a product has been developed as the costs of course correction can be high.
While researching usability is a task that requires experience and understanding human and product psychology, doing it is preferred to not doing it, where the minimum gain can be uncovering serious issues and friction with your product.
While usability testing can be highly detailed, there are generally four steps to conducting one:
- Recruiting customers/users
- Preparing the test
- Presenting and engaging with the prototype
- Analyzing, summarizing, and sharing the learning
Recruiting customers/users
If you are an established product and corporation and have an agency that can provide you with the test subjects then that’s a huge advantage. If not, then here are some options:
- If you’ve already conducted a customer discovery program, it’s best to tap into the users that you worked with
- Set up advertisements to recruit target customers/users from online and offline channels
- Reach out to your product marketing or customer success/support teams for target customer/user contacts and to reach out to them through a campaign
- Advertise a volunteer campaign on your current product — most products currently use this tactic
- Depending on your task and ask, motivate subject participation of subjects through compensation and/or by reducing frictions across their participation journey
Preparing the test
Preparing usability requires thinking in detail about the steps you and the subject have to take, preparing your thoughts and questions in advance, and minimizing biases and friction. Here are some thoughts:
- While low/medium-fidelity prototypes can bring in insights, it’s best to have high-fidelity ones to understand customers/users’ near-end product experience and reactions and also gauge value risks
- Have a full team in the meetings/interviews including product managers, designers, user researchers, and engineers to have holistic and shared learning
- Stay away from the emotional bias of falling in love with your product, let the subjects speak, listen as much as possible without interrupting, and engage with open-ended questions to uncover learning
- Dedicate one person to administer the test, another to take notes, and one other to debrief insights with after the test/interview
- Try to set up the surroundings to feel natural and comfortable to the interviewee and minimize environmental bias
- If you have a B2B product, then it could make sense to conduct the test/interview at your partners’ location
- Remote tests can be helpful, but not a replacement for in-person meetups as you can lose the customers/users’ body language and emotional reactions
Presenting and engaging with the prototype
Engagement with subjects to test a prototype’s usability can be an exhaustive task, but a crucial procedure for product teams. Below are some pointers:
- Validate that the subject qualifies for the test by asking about their pain and how they currently solve it
- When presenting the prototype let the subject know that they are only reviewing the prototype and their feedback won’t hurt your feelings; hence, the more honest that they are, the better
- Try to keep the focus of the test subject on usability and use mode over aesthetics and critique mode. Design can be changed and perfected more easily than user flow and experience
- Keep quiet, observe, and guide if/when needed. Avoid leading the test!
- If there is too much silence, ask the subject to walk you through what they are doing and/or thinking
- Observe emotional cues by reading their body language and tone to pick up frustrations, gratitude, happiness, joy, excitement, boredom, etc.
- Categorize your subject by a) the user got through the task with no problem and help b) the user experienced problems but eventually got through it, and c) the user got frustrated and quit the test. This will help you better understand the needs of various customer/user archetypes
Analyzing, summarizing, and sharing your learning
The main goal of this exercise is to gain an understanding of customers/users and friction points in the prototype. As soon as you’ve identified problems, fix them and test them on previous or new subjects. In other words, if you think a prototype, for whatever reason is obsolete, just change it, test it, and move on rapidly — don’t waste time showing the same prototype to other subjects.
After each interview, either the product manager or designer, should draft a summary email of the key learnings and send it to the larger product team. After all, interviews are completed, and if the findings are substantial, it would be a wise idea to present findings and discoveries to the larger product organization to share your learning, educate one another, and get feedback.
Testing for feasibility
When discussing feasibility concerns, we are generally looking for answers to 8 technical questions:
- Does the team have the know-how and skill sets to build the product?
- Do we have the available timelines to make the product?
- Do we have the right architecture in place to make the product?
- Will the product scale?
- Do we have all the shared components and dependencies outlined in building this product?
- Will the performance and stability of the product be acceptable?
- Do we have the testing infrastructure to quality control the product?
- Will the development costs meet our budget and investment plans?
While most of these questions for most feature and product efforts can be answered readily and easily, there are times were the engineering team and delivery managers may find it difficult to have readily available answers and require further investigation and at times developing minor feasibility prototypes such as in the case of building ML and automation technologies.
Most often, if the engineering team has been part of the product and customer discovery efforts of the product managers, especially during prototyping initiatives, then they should have already started their investigations and have answers to most of the questions. This shared learning during product and customer discovery can ease later pain in collaboration between product managers and engineers.
If you are developing a hardware product, due to the higher upfront costs compared to software, it’s crucial to have a rigorous risk assessment process to answer feasibility, usability, value, and viability questions to raise the level of confidence of the team before committing to manufacturing.
Testing for economic viability
While it’s great to have a product that customers/users will love and remain loyal to it, it’s important to have a viable business model that covers the costs to produce and sell, meets the regulatory requirements, and generates profits.
A product does not live on its own but in an ecosystem with external and internal stakeholders. While testing for value and usability tries to gauge alignment with the needs of the external stakeholders, a viable product needs to protect revenues, business reputation (e.g., regulation, business relationships, brands, etc.), employees, and customers.
1. Caring for product marketing
Your product marketing colleagues care about the following:
- Enabling sales
- Brand and reputation of the products and business
- Market competitiveness and differentiation
- Being relative and compelling to external stakeholders
- Working well with go-to-market and launch channels
A product will only be considered business viable from the perspective of a product marketing team when it addresses the above criteria. Any conflict with these priorities will result in resistance from your product marketing team. It’s important to align with product marketing before building products and to address their concerns before development.
2. Caring for sales and business development
Whether you have direct sales teams or indirect ones such as through ads, any change or product work needs to be designed and built around the strengths and limitations of your sales channels. Furthermore, sales and biz dev teams have long-term contracts and commitments to partners that may cripple product development due to agreements.
Product development needs to be aligned with sales know-how, channels, and relationships and any new proposals will require to have sales effectiveness or sales viability.
3. Caring for customer support/success
Businesses either have a high-touch or low-touch model of interacting with customers to resolve their problems (or somewhere in between). When building products, to create value for customer success and become a viable product, the product needs to take that model into account.
4. Caring for finance
Finance will need to understand your product’s costs to build, sell, and operate and to agree on a budget for your product. Finance and budgeting is often the biggest constraint that product teams need to deal with when building products. Therefore, a viable product needs to align its scope with the budget and resources available to it before development.
5. Caring for legal
Privacy and regulatory compliance concerns, intellectual property rights, and competitive issues are the common legal constraints any product will have to deal with. And product teams need to align with their legal departments before commencing development.
6. Caring for senior leadership
In general, aligning product discovery with the above 5 internal stakeholders while having proof of product value, usability, and feasibility should be enough for any member of the senior leadership team to align with your efforts for product development. Reaching out to align with senior people without having aligned with the other stakeholders is a miss.
Hope you enjoyed reading this article! :)
If so, then:
- Follow me on Medium
- Become a Medium Member
- Subscribe to hear more
- Let’s connect on LinkedIn
