Essentials in ML/AI product management

Managing ML/AI projects for Product Leaders and Managers — Part 3: Data Considerations

Data Considerations in Machine Learning Projects: Maximizing Success through Intentional Data Management

This article discusses the crucial role of data management in ensuring the success of machine learning (ML) projects. Competent product managers must understand the importance of intentional data collection, ensuring data quality, and representing the entire population to build robust and accurate models. Continuously updating data allows adaptability to changing environments. Data cleaning and accessibility are emphasized for effective model training by data science teams. The article also delves into the significance of collaboration, versioning, and reproducibility tools for seamless teamwork and efficient tracking of model performance. By prioritizing these data considerations, organizations can harness the power of ML and drive innovation in diverse applications and industries. The article also covers essential aspects of data needs, data sourcing, and data preparation for successful ML modeling, providing insights into real-world examples and best practices for addressing challenges like messy data and data silos. Additionally, feature selection methods are explored, offering an understanding of their pros and cons to streamline model complexity and reduce overfitting, ultimately maximizing the value and success of ML/AI projects.

Photo by Pietro Jeng on Unsplash

Comprehending the role of Machine Learning (ML) and Artificial Intelligence (AI) and their impact on products is increasingly crucial for competent product managers. The high failure rate of ML projects, as widely reported, is attributed mainly to factors unrelated to the models themselves.

Machine Learning (ML) is a set of methods and tools which help realize the goal of the field of Artificial Intelligence (AI). Deep Learning (DL), or the use of Neural Networks (NN) containing many layers, is a subfield of ML. Furthermore, areas such as Computer Vision, Natural Language Processing (NLP), and Recommendation Systems are sub-fields of AI that rely on ML methods and Deep Learning to deliver value

By adopting best practices in identifying ML opportunities, carefully considering key design decisions for ML systems, and implementing a disciplined approach to ML project management, product leaders and managers can greatly enhance the chances of success and significantly reduce the prevailing high failure rates.

To deliver value and achieve success in ML/AI projects, competent product managers should focus on performing five critical tasks:

1. Identify and frame problems: it is crucial to identify suitable opportunities where ML can address user/customer problems effectively. Product managers should frame these problems in a way that allows for the design of ML-based solutions
2. Organize ML projects using CRISP-DM: understanding and implementing the CRISP-DM (Cross-Industry Standard Process for Data Mining) data science process helps in organizing ML projects and coordinating team efforts efficiently
3. Grasp the data-related aspects of ML projects: product managers must clearly understand data-related considerations when building ML systems. This includes identifying data requirements, exploring potential data sources, establishing data governance and access protocols, and recognizing the importance of data cleaning and preparation before modeling.
4. Design ML systems and select technologies and tools: familiarity with the critical elements of designing ML systems is essential. Product managers should consider various factors when choosing technologies and tools for ML projects, ensuring optimal choices are made
5. Manage the model lifecycle: even after a model is released, its performance needs to be actively managed. Product managers should monitor and maintain models over time, ensuring that they continue to perform effectively in an evolving environment.

By fulfilling these tasks, competent product managers can maximize the value and success of ML/AI projects.

Photo by Taylor Vick on Unsplash

Part 3: Data Considerations

The key to success in machine learning projects lies in the effective management of data.

Intentional data collection, focusing on quality, quantity, and unbiased representation, forms the foundation of a successful model

Continuous data updates ensure adaptability to changing environments

Data cleaning and accessibility for data science teams are crucial for robust and accurate model training

Collaboration, versioning, and reproducibility tools are essential for seamless teamwork, efficient tracking of changes, and validation of model performance

By prioritizing these data considerations, organizations can harness the power of machine learning and drive innovation in diverse applications and industries.

Data Needs for Machine Learning Projects

Understanding data needs is a foundational step in machine-learning projects. Identifying essential features, obtaining accurate labels, and ensuring an adequate quantity of data are vital for building powerful and effective models. By exploring real-world examples, we recognize the importance of data in achieving success in diverse machine learning tasks. Harnessing the power of data enables us to unlock the potential of machine learning and transform industries, making groundbreaking advancements in the field of artificial intelligence.

Understanding Data Needs for Training and Prediction

For a successful machine learning project, we must recognize the dual role of data.

• Firstly, historical data is vital for training the model. This includes input features and target values (labels) from past observations. By utilizing this historical data, the model learns patterns and relationships, allowing it to make informed predictions.
• Secondly, real-time data is essential for prediction. As new data streams into the model, it generates up-to-date outputs, ensuring its relevancy and adaptability in dynamic environments.

Identifying Training Features

Selecting the right features is a critical aspect of model development. [Subject matter experts] play a key role here, offering valuable insights into the domain and suggesting relevant factors to consider when building the model. Their expertise guides us in making informed decisions about which features are likely to contribute significantly to solving the problem.

[Customers], whether internal or external, also offer valuable input. As they are the ones facing the problem, they possess a unique understanding of the contributing factors and potential solutions. Engaging in discussions with customers helps us identify additional features that may influence the model’s performance.

Furthermore, considering [temporal and geospatial characteristics] can enhance the model’s ability to capture patterns and variations. For instance, time series problems may benefit from including features such as hour of the day, day of the week, or seasonality, providing the model with valuable context.

Determining the Optimal Number of Features

The number of features in a model is a crucial factor affecting its performance. Starting with a small set of features allows us to establish a baseline model, serving as a reference for evaluating the impact of additional features. Gradually adding features and assessing their contribution helps fine-tune the model.

When in doubt about including a feature, it is advisable to try it out and evaluate its impact. Missing a key feature can severely limit the model’s predictive capabilities.

It is generally better to have more features that might logically be relevant than to miss critical ones

The Role of Labels in Model Training

Accurate labeling is fundamental for supervised machine learning tasks

Labels represent the target values the model aims to predict. The type of problem being tackled dictates the form of the label. For instance, in image recognition, labels may represent the most prominent object or multiple objects with their locations.

Creating suitable labels is vital, but obtaining them can sometimes be challenging. In scenarios like image labeling, hand labeling may be necessary, which can be time-consuming but crucial for model training.

The Impact of Data Quantity on Model Performance

Data quantity plays a significant role in model performance.

Generally, having an order of magnitude more observations than features or labels is recommended

The number of features and the complexity of relationships between inputs and targets influence data quantity requirements

Text or image-based problems with a large number of features demand a considerable dataset for effective model training.

Additionally, data quality is vital, as missing or noisy data may increase the need for a larger dataset

Real-World Examples of Varying Data Needs

To gain a deeper understanding of data requirements, some real-world examples include:

1. Iris Flower Dataset: The Iris dataset is a classic example of a relatively simple problem with only four features. Achieving high performance on this task requires a small dataset of around 150 observations
2. Chest X-Rays (Check Spurt Dataset): The Check Spurt dataset contains 224,000 labeled chest X-ray images, making it a more complex problem with numerous features. This complexity demands a substantial dataset to build a reliable model
3. ImageNet Dataset: ImageNet, with its vast collection of 14.2 million labeled images across thousands of object classes, presents a highly complex image recognition challenge. A massive dataset is required to train a model effectively for this task
4. Gmail: Google’s Gmail Smart Reply product was trained on a dataset of 238 million observations, while Google Translate was trained on trillions of observations as a large number of features and intricate relationships in language translation demand an extensive dataset to achieve accurate predictions.

Sourcing Data for Machine Learning Models

Sourcing the right data for machine learning models is fundamental to building successful AI systems. Internal data, customer-generated data, and third-party data offer valuable insights to enhance model performance and user experience. Adopting best practices for data collection, avoiding bias, and ensuring data representation are essential for the success of AI projects. The flywheel effect of user-generated data allows for continuous improvement and expansion of AI applications while addressing the cold-start problem, empowering recommendation systems to cater to new users effectively. As AI technologies continue to evolve, the significance of data-driven solutions will remain central to driving innovation and transforming industries.

Understanding Data Sources

There are different sources from which product teams can obtain data for their machine-learning models.

• Internal data, generated within the organization, includes data from log files, user data collected from websites, operational data, or data from machinery, and can be found within ERP systems
• Customer-generated data is another valuable source, derived from deployed sensors, user behavior on websites, votes, rankings, ratings, or online forms
• External third-party data sources can be useful for specific modeling problems, such as weather data for forecasting or demographics data for user profiling

Best Practices for Data Collection

Collecting data intentionally is a vital best practice. Rather than amassing all available data, it is essential to focus on what is truly necessary for the model. By doing so, we can manage storage and processing costs and address privacy and ethical concerns. Data collection should also be mindful of avoiding bias, ensuring that data sources are diverse and representative of the population being modeled.

Documentation and Metadata

Documenting the data collection process is crucial for maintaining transparency and ensuring continuity within the team. Metadata, or data about the data, provides valuable context for understanding the attributes and relationships in the dataset. Proper documentation aids in avoiding confusion and difficulties when revisiting the data later in the development process.

User Data as a Valuable Source

User data is a popular and rich source of information for building models. Websites can collect data through forms, user behavior analysis from website logs, votes, rankings, or ratings provided by users. The ideal approach is to collect data non-obtrusively, seamlessly integrating data collection into the user’s workflow. Providing benefits to users through the data collection process, such as personalized recommendations or improved services, encourages participation.

Creative Example of User Data Utilization

The CAPTCHA product from Google serves as a creative example of collecting user data. While verifying users as humans through image selection, Google simultaneously collects valuable labeled data, which can be used for training machine learning models, such as image recognition systems.

The Flywheel Effect of User-Generated Data

The flywheel effect describes a cycle where user interactions generate data that feeds into AI systems, leading to improved system quality and more opportunities for AI applications

Amazon is a prime example of employing user-generated data to refine its system as it uses customer data to reorder product listings, provide personalized recommendations, and identify commonly purchased items to offer customers relevant suggestions.

TikTok’s algorithm-friendly design — the future of video and social networks

Addressing the Cold-Start Problem in Recommendation Systems

The cold-start problem arises when a new user engages with a recommendation system without any historical data on their preferences. To overcome this, systems may use heuristics-based approaches initially, or incorporate a calibration step to gather user data and train simple machine learning models. As users continue to interact with the system, it improves and personalizes the recommendations.

Data Governance and Access: Breaking Down Data Silos and Enhancing Access and Collaboration

Breaking down data silos is a transformative step toward unleashing the full potential of data-driven insights within organizations. A combination of cultural change, technological improvements, and data stewardship enables seamless data access, fosters cross-functional collaboration, and empowers data scientists and business users to make informed decisions based on real-time, accurate, and comprehensive data. As more organizations embrace the data-driven culture and address the challenges of data silos, they position themselves to thrive in an increasingly data-centric world, driving innovation and achieving business excellence through machine learning and AI.

The AI Factory and its Components — The Economics and Business Logic of AI

Understanding Data Silos

Data silos are barriers that impede data accessibility, resulting from different departments using disparate systems and schemas for data collection and storage

The consequences of data silos include limited access to data, increased difficulty in understanding the available data, and reduced opportunities for cross-functional collaboration. For organizations new to machine learning, focusing on breaking down data silos is vital before initiating complex modeling projects.

Cultural Change

The process of breaking down data silos begins with a cultural shift within the organization. Having an executive sponsor or a high-ranking individual championing open access to data is crucial. Encouraging different departments to collaborate and centralize data for open access requires incentives and education. Building a data-driven culture that values cooperation and knowledge sharing across teams paves the way for more effective data utilization.

Leveraging Technology

While cultural change is essential, technology also plays a critical role in dismantling data silos. Organizations need to consider centralizing their data in a data warehouse to ensure data is accessible and organized. Deciding how users can query the data and retrieve relevant information is equally vital. Implementing user-friendly interfaces and tools, such as Hive, which allows users to query data using familiar SQL queries, can dramatically enhance data accessibility for non-technical users.

Data Stewardship and Access

Effective data stewardship is essential for maintaining data quality and cleanliness. Assigning responsibility for data maintenance and organization helps ensure data is reliable and accurate. To enable users throughout the organization to access the data they need without bureaucratic delays, implementing a data catalog or data map is valuable. This facilitates data discovery and simplifies the access process, empowering users to retrieve data efficiently and effectively.

How We Improved Data Discovery for Data Scientists at Spotify - Spotify Engineering

Dealing with Messy Data in Machine Learning Projects

Messy data poses significant challenges in machine learning projects, impacting model quality and performance. Understanding the different types of missing data and outliers is key to making informed decisions about data handling strategies. By employing appropriate data imputation techniques and being cautious with outlier treatment, machine learning practitioners can enhance the reliability and accuracy of their models, ultimately driving the success of their projects in various domains.

The Dilemma of Missing Data

Missing data is a pervasive issue in machine learning projects, and it can arise from various sources. Users might fail to provide certain data fields in web forms, or manual data entry could lead to mistakes and omissions while sensor data may suffer from issues such as power or communication failures, disrupting data collection. Understanding the types of missing data is crucial, as it influences subsequent decision-making.

There are 3 types of missing data:

• Missing Completely at Random (MCAR): In this scenario, the missing data exhibits no discernible pattern or association with other attributes. For example, sensor data missing due to random power outages fall under this category. MCAR data is generally less concerning, as it doesn’t introduce significant bias into the models.
• Missing at Random (MAR): Here, the probability of data being missing relates to another feature in the dataset. For instance, in a medical survey, males may be less likely to answer depression-related questions than females. MAR data can lead to biased models, making it a more significant concern.
• Missing Not at Random (MNAR): In MNAR situations, the missing data’s probability is directly linked to the feature itself. For example, if product ratings are skewed towards negative values, users with positive reviews may be less likely to provide ratings. MNAR data has a high potential to introduce bias into models and demands careful handling.

The Impact of Missing Data Types on Machine Learning Bias — Understanding the different types of missing data is crucial in machine learning projects as it directly influences the potential bias in models. This table categorizes the three types of missing data, from no discernible pattern to direct links with other features, along with their respective levels of bias potential.

Addressing Missing Data

To tackle missing data, several strategies can be employed:

• Data Removal: In cases of substantial data collection, omitting rows or feature columns containing missing data may be a viable option. However, this approach should be employed judiciously to avoid information loss
• Flagging Missing Data: In some instances, knowing that data is missing can serve as a relevant feature in modeling. Instead of removing it, assign a special value to signify missing data.
• Data Imputation: Missing data can be replaced with meaningful values. This can involve using the mean or median of the feature, forward or backward filling, or even inferring the missing value using regression models.

Taming the Outliers

Outliers are data points that significantly deviate from the rest, either in feature values or target values. Dealing with outliers is critical, as they can wield disproportionate influence on models and distort the model’s behavior.

• Identifying Outliers: Various methods can be used to detect outliers, such as statistical tests and visualizations. Visualizations, in particular, provide a quick way to spot potential outliers. Scatter plots and boxplots are commonly used to visualize data and identify outliers.
• Addressing Outliers: When outliers are identified, it’s essential to ascertain their nature. Some may represent real data, while others could be errors or anomalies. For erroneous outliers, removal or adjustment can be considered. However, care should be taken not to automatically remove all outliers, as they may hold critical insights, especially in extreme events.

Preparing Data for Machine Learning Modeling

Data preparation is a crucial stage in any machine learning project. From cleaning and exploring data to engineering and selecting features, each step plays a significant role in determining the success and accuracy of the model. By understanding and effectively executing these essential data preparation processes, machine learning practitioners can build robust models that deliver powerful insights and predictions across a wide range of applications.

Data Cleaning: Taming the Messy Data

Data collected from diverse sources may be plagued with missing values and outliers. Missing data can occur due to various reasons such as user omissions, errors during data entry, or equipment malfunctions in sensor data collection. Data cleaning involves removing or filling in these missing values and identifying and handling outliers that can distort the modeling process.

Exploratory Data Analysis (EDA): Unveiling the Hidden Insights

EDA serves as a vital preliminary step to comprehend the underlying patterns and relationships within the data. Utilizing statistical methods and visualization techniques like scatter plots and correlation matrices, EDA helps to identify trends, distributions, and associations within the dataset. Understanding the data better enables the selection of relevant features that have a significant impact on modeling outcomes.

Feature Engineering: The Building Blocks of Successful Modeling

Feature engineering involves the creation and selection of features that will serve as the inputs to our machine learning model. Features can be natural attributes of the data, like temperature or humidity, or engineered features, such as converting text data into numerical values for analysis. Crafting the right set of features is crucial for achieving effective model performance.

What is Feature Engineering — Importance, Tools, and Techniques for Machine Learning

Feature Selection: Streamlining Complexity and Reducing Overfitting

With an extensive feature set, feature selection becomes necessary to streamline model complexity, mitigate overfitting, and improve training time. Three primary methods for feature selection are:

• Filter Methods: Employ statistical tests to identify correlations between features and target variables, making it easy to identify and eliminate irrelevant or non-contributing features early in the modeling process.
• Wrapper Methods: Evaluate multiple models with different subsets of features to determine which features contribute most to the model’s quality. Though effective, wrapper methods can be computationally expensive.
• Embedded Methods: Utilize model-specific characteristics like feature importance in decision trees or random forests to identify the most valuable features during the model-building process.

Feature Selection Methods — an overview of three primary feature selection methods used in machine learning projects. Feature selection is essential for reducing model complexity, preventing overfitting, and improving training time. Each method has its own set of advantages and disadvantages, offering different trade-offs in terms of computational efficiency, ability to handle large datasets, consideration of feature interactions, and dependence on model hyperparameters. Understanding these methods can help data scientists make informed decisions in selecting the most suitable approach for their specific modeling tasks.

Data Formatting: Preparing Data for Modeling

The final step in data preparation involves formatting the data to be compatible with machine learning algorithms. This includes scaling numerical data to bring all features to the same order of magnitude and converting categorical variables into numerical codes to be used as model inputs.

Reproducibility and Versioning in Machine Learning

Reproducibility and versioning are essential cornerstones of successful machine learning projects. They enable efficient debugging, knowledge transfer, collaboration, and credibility while supporting model evolution and seamless deployment. By embracing these principles and best practices, data scientists and researchers can build robust, reliable, and impactful machine learning models that deliver valuable insights across various domains and applications.

The Importance of Reproducibility

• Debugging and Troubleshooting: In the ever-changing landscape of machine learning projects, reproducibility helps pinpoint the sources of errors or performance issues that may arise during production runs. By understanding the data transformations and modeling processes, teams can identify and address potential problems more effectively.
• Knowledge Transfer and Collaboration: Team dynamics in machine learning projects often involve the sharing of models and pipelines between team members. Properly documented and reproducible projects facilitate seamless knowledge transfer and collaboration, ensuring that others can reproduce results even in the absence of the original developer.
• Credibility and Peer Review: For academic researchers or professionals in industries with strict compliance requirements, reproducibility is essential to establish credibility and allow peer reviews of model results. Reproducible experiments and results increase confidence in the findings and contribute to the credibility of the model.

Best Practices for Reproducibility

• Proper Documentation: Thoroughly document the functionality of models and data pipelines, including dependencies between data, code, and models. Clear and comprehensive documentation makes it easier for team members to understand, replicate, and maintain the project.
• Data Lineage: Data lineage involves tracing data from its raw sources through the various stages of processing and transformations to its final form used for modeling. Data lineage is essential for debugging and ensuring the integrity and quality of the data.
• Versioning: Proper versioning of code, data, and models allows for tracking changes and iterations throughout the development process. It helps avoid repeating mistakes, facilitates easy reversion to previous versions, and allows for champion-challenger model testing.

Data Lineage: Tracing the Data Journey

Data lineage is the process of tracking data from its raw source to its final form used for analysis and modeling. Benefits of data lineage include:

• Debugging Support: Understanding the transformation steps data undergoes during the pipeline helps identify potential sources of errors or discrepancies in model performance.
• Simplified Data Migrations: When moving data between systems, data lineage ensures a smooth and accurate transfer, reducing the risk of data loss or corruption.
• Trust in Data Quality: Transparency in data lineage inspires trust in the data’s accuracy and reliability, encouraging users to rely on the data for their applications or models.

Model Versioning: Keeping Track of Model Evolution

Model versioning involves tracking the different iterations of a model as it progresses through the development and deployment stages. Key benefits include:

• Learning from Iterations: Tracking model versions helps data scientists learn from each iteration, identifying what works and what doesn’t, thus refining and improving the model’s performance.
• Champion-Challenger Model Testing: By comparing multiple model versions in parallel, organizations can select the best-performing model for production, ensuring optimal model deployment.
• Reverting to Previous Versions: In case of performance degradation or issues with the production model, the ability to revert to previous versions ensures continuity and minimizes downtime.

Implementing Reproducibility and Versioning

Implementing reproducibility and versioning can be achieved using commercial software, machine learning platforms with built-in versioning features, or open-source model versioning packages. Detailed documentation, tracking data lineage, and maintaining version histories are essential practices to ensure the long-term success and sustainability of machine learning projects.

Hope you enjoyed reading this article! :)

If so, then:

• Follow me on Medium
• Become a Medium Member
• Subscribe to hear more
• Let’s connect on LinkedIn

Join Medium