As artificial intelligence (AI) becomes more prevalent and every industry races to develop AI solutions to advance their use cases, practical challenges have arisen around production deployment.
In my previous blog post , I described a process for taking machine learning (ML) experiments to production deployments. In this follow up post, I outline seven AI industry trends that help users simplify and scale the overall ML lifecycle. We describe each trend, discuss why it matters for operational ML and what factors should be considered as a business decides to exploit a trend to accelerate or improve their operational ML practice.
Figure 1 shows a typical machine learning (ML) lifecycle. Over time, the cycle repeats as the ML function is further optimized relative to the business need.
The first challenge of many ML initiatives is finding an acceptable dataset. Data marketplaces attempt to address the shortage of datasets, particularly in key fields such as Healthcare  and IoT [10,] by providing platforms where individuals can share their data and companies can consume the data for AI and analytics purposes. The marketplace platform guarantees security, privacy and also provides an economic model to incentivize participants . Data marketplaces can provide a rich source of otherwise unavailable data, and the marketplace can provide data provenance and lineage information that is later needed to manage the data and ensure quality.
Yet another angle to address the dataset shortage is the marketplace of synthetic datasets. Machine learning technology advances have demonstrated that ML itself can be useful to generate realistic datasets to train other ML algorithms, particularly in the deep learning space . Synthetic data has been lauded for its potential to level the AI playing field for smaller companies relative to larger organizations who have access to massive data sets. Synthetic data can range from anonymized versions of actual datasets to extended datasets generated from real data samples, to simulation environments such as virtual environments used to train autonomous vehicles .
Good data sets are scarce, but good datasets which are labeled are even more scarce. To address this problem, a marketplace has emerged for data labeling, focusing frequently on specific data types (such as objects in an image). Some of the labeling comes from human labelers coordinated across geographies and managed via coordination software . Companies are innovating in this space with hybrids of human and ML-based labeling , a sub-trend that has the potential to reduce the cost of human-only labeling. Other innovations in this category include services which enable businesses to interact with labeling service providers directly .
Once a suitable dataset is found and labeled, the next challenge is finding a good algorithm and training a model. Automated machine learning (AutoML) technologies automate the process of algorithm/model selection and tuning, taking an input dataset and running a large number of training algorithms and hyperparameter options to select a final model that is recommended for deployment. Related to AutoML (and frequently provided within AutoML products) is the automation of feature engineering with technologies such as Deep Feature Synthesis . AutoML software can also sometimes perform bias detection for the input dataset. Some AutoML solutions are SaaS offerings while others are downloadable software [4,5] that can run in a containerized form on cloud or on-premise environments.
For those who may be developing their own model, containers are a well-established design pattern for production deployments since they enable any training or inference code to be run in a well-defined environment that is portable and scalable. Orchestration tools such as Kubernetes further enable scale and flexibility with container-based ML. However, assembling a container can be a challenging task since dependencies have to be resolved and the entire stack tuned and configured. Pre-built container marketplaces address this problem, providing pre-configured containers with necessary libraries pre-installed and configured, particularly for complex environments such as GPUs  .
If you do not want to build or train your own model, there are model marketplaces. Model marketplaces enable customers to buy pre-built algorithms and sometimes trained models [7,8]. These can be useful for the following use cases. (a) the use case is sufficiently generic so there is no need to train a custom model and outfit training/inference code into a custom container, (b) mechanisms like Transfer Learning can be used to extend and customize base models, or (c) the user does not have enough training data to build their own model. In Model marketplaces the significant work of processing the data and training a good model can be offloaded, enabling the user to focus on the other aspects of operationalization. That said, a key challenge with model marketplaces is sifting through the content to find assets that suit your needs.
Finally, for common use cases that exist across businesses, application level AI services can remove the need for the entire operational ML lifecycle. Rather than creating models, training and deploying them, one can subscribe to an end service that performs the AI task. Application level AI services exist for vision, video analytics, Natural Language Processing (NLP), form processing, Natural Language Translation, Speech Recognition, Chatbots, and other tasks [18, 19].
All of the above trends enable users to simplify or accelerate one or more stages of the operational ML lifecycle, either by offloading, reusing pre-built items, or via automation of particular stages. Given how iterative ML processes can be (for example, training usually includes tens to hundreds of experiments), automating these processes can result in more trackable, reproducible, and manageable workflows. Outsourcing these tasks can be even easier, particularly if hardened models and algorithms (that have been tested in many environments besides your own) can be leveraged for basic tasks.
That said, there are several factors to consider before using one of these services in your environment:
Not all trends are applicable to all use cases. The most generally applicable trend is AutoML which spans a wide range of applications. Similarly, Model Marketplaces have a very wide range of models and algorithms available. Data Marketplaces and Synthetic datasets tend to be specific to classes of use cases and Pre-Built Containers can be specific to different hardware configurations (like GPUs) which then lend themselves to specific usages. Many data labeling services are also usage specific (such as for image classification and form reading), but some consulting firms do offer customized labeling services. Finally, the end-to-end AI services are extremely use case specific.
As more ML gets deployed, ordinary human fears about black box AI systems are manifesting in trust concerns and increased regulations . To benefit from AI, businesses have to consider not just the mechanics of production ML but also managing any customer and/or community concerns. Left unaddressed, these concerns can materialize in customer churn, corporate embarrassment, brand value loss, or legal risk.
Trust is a complex and expansive topic, but at its core, there is a need to understand and explain ML and feel confident that the ML is operating correctly, within expected parameters and free from malicious intrusion. In particular, the decisions made by the production ML should be explainable - i.e. a human-interpretable explanation must be provided. This is becoming needed in regulations such as the GDPR’s Right to Explanation Clause . Explainability is closely tied to fairness - the need to be convinced that the AI is not accidentally or intentionally rendering biased decisions. For example, AI services such as Amazon’s Rekognition have been called out for concerns of bias .
Since virtually all of the trends above involve offloading or “outsourcing” some aspect of the ML lifecycle to a third party or automated system, additional awareness needs to be taken at each stage to make sure that the end production lifecycle can deliver the core tenets for trust. This includes knowing what algorithms are deployed, whether the datasets used to train them are free of bias, etc. These requirements do not change the lifecycle itself, but additional diligence needs to be paid to ensure proper lineage tracking, configuration tracking, and diagnostic reporting.
Regardless of where the components of the ML lifecycle came from, your business will be responsible for managing and maintaining the health of your ML services through their lifecycle (with the possible exception of entirely outsourced services as in Trend 7). If so, it is important that your data scientists and engineers understand the models that are being deployed, the datasets that were used to train them, and the expected safe operating parameters of those models. Since many of the services and marketplaces are nascent, there is no standardization at present. The consumer is responsible for understanding the services that they use and ensuring that the services can be adequately managed in conjunction with the rest of the lifecycle.