Essentials in ML/AI product management
Managing ML/AI projects for Product Leaders and Managers — Part 1: Identifying the right opportunities
Identifying and framing problems is crucial to finding suitable opportunities where ML can address user/customer problems effectively. Successful ML product managers follow a process of problem screening, demand validation, and progressive development
Comprehending the role of Machine Learning (ML) and Artificial Intelligence (AI) and their impact on products is increasingly crucial for competent product managers. The high failure rate of ML projects, as widely reported, is largely attributed to factors unrelated to the models themselves.
By adopting best practices in identifying ML opportunities, carefully considering key design decisions for ML systems, and implementing a disciplined approach to ML project management, product leaders and managers can greatly enhance the chances of success and significantly reduce the prevailing high failure rates.
To deliver value and achieve success in ML/AI projects, competent product managers should focus on performing five critical tasks:
- Identify and frame problems: it is crucial to identify suitable opportunities where ML can address user/customer problems effectively. Product managers should frame these problems in a way that allows for the design of ML-based solutions
- Organize ML projects using CRISP-DM: understanding and implementing the CRISP-DM (Cross-Industry Standard Process for Data Mining) data science process helps in organizing ML projects and coordinating team efforts efficiently
- Grasp the data-related aspects of ML projects: product managers must clearly understand data-related considerations when building ML systems. This includes identifying data requirements, exploring potential data sources, establishing data governance and access protocols, and recognizing the importance of data cleaning and preparation before modeling.
- Design ML systems and select technologies and tools: familiarity with the key elements of designing ML systems is essential. Product managers should consider various factors when selecting technologies and tools for ML projects, ensuring optimal choices are made
- Manage the model lifecycle: even after a model is released, its performance needs to be actively managed. Product managers should monitor and maintain models over time, ensuring that they continue to perform effectively in an evolving environment.
By fulfilling these tasks, competent product managers can maximize the value and success of ML/AI projects.
Part 1: identifying and framing problems for ML/AI
Identifying opportunities for ML/AI
One of the primary reasons for the low success rate of ML projects within data science teams is the selection of incorrect problems to work on. Choosing the right problems that can be effectively solved through the application of ML is a challenging task.
It is important to prioritize problems that meet the following criteria:
- Add business value: select problems that provide tangible benefits and value to users/customers, aligning with their needs and expectations. Solving these problems should have a positive impact on the business or organization
- Fit for ML: ensure that the chosen problems are well-suited for ML solutions. Not all problems can be effectively addressed using ML techniques, so it is crucial to assess whether ML is the appropriate approach for solving the problem at hand
- Validation with best practices: establish a framework to validate the solutions derived from ML models using industry best practices. This ensures that the proposed solutions are reliable, accurate, and aligned with established standards and methodologies.
- Jump-start development potential: identify problems that can be jump-started into the development phase by leveraging heuristics and baseline models. This approach allows for initial progress and lays a solid foundation for further refinement and improvement of the ML models
By adhering to these guidelines and selecting problems that meet these criteria, data science teams can increase their chances of success in delivering ML projects and maximize the value they provide to users and customers.
The wrong way to start an ML project
ML projects often experience high failure rates, and while technical factors can play a role, many failures stem from non-technical reasons. The initiation of such projects can be driven by various factors, including:
- FOMO factor: Executives may initiate ML projects out of the fear of missing out (FOMO). They perceive that other companies are already leveraging AI and feel compelled to follow suit without a clear understanding of how ML can truly benefit their specific business needs
- Mismanagement and lack of specialization: Sometimes, projects are initiated based on a top-down directive without a proper understanding of the requirements or the expertise needed. Lack of empowering specialization and mismanagement of the project can hinder its success
- Getting caught in the sales funnel of partners: In certain cases, ML projects are pursued simply to align with the offerings of third-party partners or vendors. The team seeks places to incorporate AI without fully evaluating the suitability or value of the technology in the specific context
- Lack of experience and diligence in working with ML: Some teams jump straight into building and tuning ML models without sufficient experience or a comprehensive understanding of the underlying concepts and best practices. This lack of diligence and expertise can lead to suboptimal outcomes.
To increase the likelihood of success, it is important to address these factors by promoting a better understanding of ML’s potential, encouraging specialized expertise, ensuring projects align with business needs, and prioritizing thorough planning and due diligence before diving into model development and tuning.
How ML/AI project should start
For ML projects to succeed, project managers need to address three screening questions to help identify good ML opportunities:
- Value creation and demand: is there a need out there? Does somebody care about the solution to the problem?
- Feasibility: can ML help solve this problem?
- Viability: does it make sense to solve the problem? What is the “ROI”?
Identifying demand and potential value to be created
To identify demand and the potential value to be created, product managers should engage with users/customers in meaningful ways, and there are three strategies they can use to that end:
- Listen to their pains: conducting one-on-one interviews or focus groups with a small group of 6–8 individuals allows product managers to actively listen to users/customers. Through prompts and discussions, users/customers can express their challenges, and by listening to each other, valuable insights can be gained. The objective is to identify and take note of pain points that make their lives difficult
- Identify current solutions: recognize that all user/customer pains are currently being addressed, albeit with suboptimal solutions. Understanding how users/customers are currently solving their problems and identifying the gaps in those solutions is crucial. This analysis helps in determining the potential value that can be created by providing better solutions
- Observe them in context: field observation or shadowing provides an opportunity to observe users/customers in their natural environments and during their regular activities. This method enables the identification of latent or subconscious pain points and problems that users/customers may not explicitly express during interviews or discussions. By immersing themselves in the context, product managers can gain deeper insights into user/customer needs
Assessing feasibility
When considering feasibility in ML projects, there are two important factors to consider:
- Problem feasibility: it’s important to understand what is easy, hard, and impossible for ML to solve, taking into account that the spectrum of easy, hard, and impossible changes over time and as the state of the art with ML/AI evolves. For example, advancements in Generative AI have opened up opportunities that were previously considered impossible
- Data availability: once it is determined that the problem is feasible for ML, product managers need to evaluate whether they have access to, or can collect, sufficient and meaningful data which is essential for effectiveness in training ML models. Product managers must consider the availability, quality, and relevance of the data required for their problem. If the necessary data is not accessible, the feasibility of the project may be compromised
Measuring viability
To ensure a viable business impact for ML projects, product managers need to think in terms of a 2-by-2 matrix framework to prioritize projects with the following axes:
- The feasibility or cost to deliver a solution
- The business impact or value quantifies benefits in numerical terms, such as increased revenue, cost savings, improved efficiency, customer satisfaction, etc.
Working on problems with low business impact regardless of whether they’re difficult or easy to deliver is not a good place to invest in.
Working on projects that have high business impact, but are very difficult to solve could be worth doing and the benefit and cost tradeoffs need to be accurately measured.
However, problems that are highly feasible and can have a high business impact are projects product managers should certainly pursue as the competencies and confidence exist in the organization and the users/customers are willing to pay for it.
Validating product ideas through experimentation
Once a project is identified and selected, the next step is to brainstorm potential solutions. At this stage, the product team may have limited information about what the solution should look like. To effectively narrow down the possibilities and identify solutions that have the potential to create value, the team employs a feedback-driven experimentation approach known as the Scientific or Hypothesis-driven Problem Solving method.
- Formulate a hypothesis about the solution: these are conceptual designs of solutions and are informed by the team’s understanding of the problem and their hypotheses about what might work
- Validate the hypothesis with demand: present users/customers with the conceptual solutions and get their feedback. The number of users/customers can differ based on the B2B or B2C context of business, however, the team should feel that there is significance in their findings and applicable and scalable to the large target population
- Assess insights and decide to continue or pivot: with feedback accumulated, the product team needs to analyze their insights and decide to continue and refine the solution concept or pivot and test new ones. They evaluate whether the hypothesis is validated, considering factors such as user satisfaction, usability, market fit, potential business impact, etc.
- Refine and repeat: repeat the process and run another experiment. Over time, teams should be running many such experiments, gathering feedback, and refining concepts so that eventually a solution is discovered that resonates with users/customers
The role of prototypes and mockups in testing hypotheses
Visual presentations of concepts and solutions can elicit valuable user/customer feedback and enhance the learning process. When concepts are presented visually through mockups and prototypes, it becomes easier for people to react and provide input compared to written or verbal descriptions. Starting with minimal designs and working prototypes allows for quick iterations and gathering feedback from users/customers.
However, it’s important to note that prototypes are representations of the product management and design teams’ vision, and their feasibility should be verified by the engineering team. In this stage, it is crucial to involve senior technical leads in interactions with users/customers to address any feasibility concerns and ensure alignment between design and implementation.
Going for development
As product teams engage further with users/customers on prototypes, they need to plan for development. This planning should begin when the following milestones have been achieved:
- The validity of the problem is confirmed
- Current solutions have been identified and their gaps outlined
- Convergence to a potential solution(s) has been realized
- The technical feasibility of the solution is established
Once these milestones have been reached and the necessary groundwork has been laid, the ML product can be staged for development. This ensures that the product team has a solid understanding of the problem, the desired solution, and the technical feasibility, enabling them to proceed with the development process in a structured and efficient manner.
The benefits of ML in products
ML/AI is complex and has a high level of technical risk and therefore should only be utilized when it can add significant value to users/customers and the business.
While FOMO is a terrible reason to apply ML to products, there are generally three good reasons why ML/AI can add value to a product:
- Bring automation
- Develop the power to predict
- Deliver enhanced personalization
Automation
Automation means eliminating repetitive and tedious work for users/customers and/or staff resulting in reduced operating costs or improved quality of service. A distinction here is automation versus augmentation where ML can enable both.
While automation refers to the replacement of machines for what humans would otherwise do, augmentation helps to support humans in their activities. For example, an ML classifier that diagnoses diseases based on chest X-rays could potentially be a risky product if it were to be fully automated resulting in the removal of the human radiologists from the diagnosis process, while, augmenting human radiologists by providing triage services using ML in areas where there’s a lack of human radiologists can be extremely helpful.
In addition, automation can also come with some important lags including:
- Lack of adaptation to the environment. ML algorithms generally struggle to adapt to major changes in the environment and with every environmental change, algorithms will need to be updated
- Lack of a sense of ethics. Humans, as social species, have certain ethical standards in what they do and how they make decisions, and these standards are not embedded in ML algorithms, and therefore automating decisions where ethics come into play needs to be treated carefully
- Lack of accountability. When machines make decisions or perform tasks, pinpointing accountability becomes a challenge, especially when things go wrong. Therefore accountability is a major risk of automation using ML
Prediction
ML algorithms have proven to be highly effective in identifying patterns and making predictions from large historical datasets, providing valuable support for decision-making. However, it’s important to be aware of the risks associated with ML prediction:
- Changes in the environment: ML algorithms are trained on historical data, and as the environment changes, they may struggle to adapt and make accurate predictions. It’s crucial for the owners of ML algorithms to continuously evaluate and monitor changes in the data and adjust the algorithms accordingly to maintain their effectiveness
- High cost of prediction errors: in certain applications, such as predicting employee success or identifying high-risk individuals, the cost of making incorrect predictions can be significant. If ML algorithms are used to make critical decisions with high stakes, the consequences of false positives or false negatives can be detrimental to both the company and the individuals involved. Careful consideration and evaluation are necessary to mitigate these risks
Personalization
ML algorithms play a significant role in delivering value through personalization by leveraging user data and preferences which can in effect result in:
- Enhanced user experiences: ML algorithms can analyze vast amounts of user data, including browsing history, previous interactions, and demographic information, to understand individual preferences and interests. This enables personalized recommendations, content suggestions, and tailored experiences that align with users’ specific tastes and preferences
- Improved engagement and satisfaction: personalization through ML algorithms helps create a more engaging and satisfying user experience by delivering content, recommendations, or suggestions that are highly relevant to each user resulting in increased user engagement, exploration, and overall satisfaction
- Efficient decision-making: ML algorithms analyze user data to make predictions and decisions in real-time such as on e-commerce platforms to personalize product recommendations based on users’ browsing and purchase history, leading to more efficient and targeted decision-making processes
ML algorithms can be built to be adaptive to continuously learn and adapt to users’ behaviors and preferences over time by analyzing feedback, interaction patterns, and response data, refining their models, and improving the accuracy and relevance of personalization over time.
The high level of value creation and the adaptability of personalization algorithms have brought significant personalization impact across various domains including media streaming platforms (Netflix, YouTube), online shopping, social media platforms, e-learning platforms, and more.
ML vs. Heuristics and the transitions state
Before embarking on building ML solutions, product teams should carefully evaluate whether ML models and algorithms are truly necessary for solving the problem, or if simpler heuristics and business rules would suffice.
This assessment can be done through a costs and benefits analysis. Heuristics offer certain advantages, such as
- Being easy to create and maintain
- Requiring minimal computational costs compared to training and running ML algorithms, and
- Being straightforward to interpret since they consist of a set of simple business rules
On the other hand, ML algorithms provide their own set of benefits:
- They often outperform simple heuristics
- Can automatically adapt and evolve through retraining using new data, and
- Are suitable for a wider range of problems, including those involving Big Data with multiple sources and features. While manually developing heuristics for such complex data can be challenging or nearly impossible, ML algorithms excel in tasks such as analyzing images or identifying and locating objects within them
The recommended approach for starting ML projects is:
- To initially rely on heuristics, as they may be sufficient to solve the problem at a satisfactory level for users/customers. By doing so, a baseline performance can be established, which serves as a reference point during the transition to building ML algorithms
- The next step involves transitioning to ML by starting with simple and interpretable algorithms, such as linear regression, logistic regression, or decision trees. These algorithms help establish an ML baseline that can be compared to the heuristic baseline. As the project progresses, the team can gradually develop more complex ML algorithms and assess their performance relative to the baseline
During the transition from heuristics to ML algorithms, two crucial questions need to be addressed.
- First, is ML necessary to achieve a satisfactory level of performance, or have satisfactory results already been obtained using simple heuristics? If the current level is deemed satisfactory, there may be no need to further develop ML models.
- Second, if ML models do provide a performance boost, are the costs associated with applying ML justified by the improvement gained? It is essential to consider the technical complexity, risks, and increased computational costs that come with ML and weigh them against the potential performance benefits
Summary
Although it may seem straightforward, finding suitable problems to solve using ML is a challenging task that holds significant importance because without addressing the right problems, subsequent efforts become irrelevant. ML product managers who achieve success follow a process of problem screening, demand validation, and progressive development.
To identify promising ML opportunities, product managers can apply three screening criteria as a general guideline:
- Is there a clearly defined problem?
- Can ML be utilized effectively to solve this problem?
- Will the proposed solution add tangible business value?
Once a potential ML opportunity is identified, product managers must validate their ideas before proceeding with product development. This involves conducting numerous small-scale experiments and tests of the proposed solution concepts with users, actively seeking their feedback, making adjustments, and iterating on the design.
When a validated idea moves into the development stage, it is advisable, to begin with the use of heuristics as a starting point. These heuristics may sufficiently address users’ needs and serve as a reasonable baseline for evaluating performance as the ML project progresses. By starting with heuristics, product managers can gauge the effectiveness of subsequent ML work against the established baseline.
Hope you enjoyed reading this article! :)
If so, then:
- Follow me on Medium
- Become a Medium Member
- Subscribe to hear more
- Let’s connect on LinkedIn
