What is Model Monitoring?
Model Monitoring provides information on the quality of machine learning models over time, alerting solution teams to potential performance degradation, especially for models deployed in production settings. Model quality is defined by looking at accuracy metrics for a model and understanding how they change over time.
Companies want to have the highest quality models possible in production, and model monitoring is a tool that enables them to do so. Tracking the quality of a model over time allows one to plan for the eventual retirement of a model once its prediction quality degrades past an acceptable point. This makes model monitoring an important task for teams looking to maximize the business value of their production models.
Model monitoring is also valuable when using a champion model in a machine learning solution. A champion model is used when multiple models are trained and deployed to solve the same business problem. At any time, only the model with the highest accuracy metrics is used for scoring, which is then called the champion model. Model monitoring enables easy determination of which model is the champion at a given time. This approach improves upon the prediction quality that is possible when a single model is deployed for a business problem.
Why does model degradation happen?
There are two main causes of model performance degradation, data drift, and concept drift. Data drift is a change in the makeup of the data on which the model is predicting. This is a common occurrence, as a model is trained on data that has a certain statistical profile, but at some point, that data may change. The scoring data being different from the training data will likely cause the prediction accuracy to suffer. This cause of model degradation can be easily identified by pairing data monitoring with model monitoring. Data monitoring is a similar concept to model monitoring and monitors for data drift.
When model monitoring detects model drift, your next step can be to check your data monitoring results on the scoring data for drift. If data drift has occurred, it has most likely contributed to the model drift. Your solution team can then dig into data drift results to understand which specific data changes caused the model drift. But what if we find that our model has drifted, and no data drift has occurred?
It is possible for model drift to occur when data drift is absent. When this is the case, concept drift is generally the cause. Concept drift happens when what defines a correct prediction changes over time, even if the profile of the input data stays the same. For example, a bank may have a definition of what constitutes a good borrower to whom they would be willing to give a loan. A year later, macroeconomic conditions may have improved. Because of this, the bank’s definition of a good borrower will have changed, as they are more willing to give out loans now. The borrowers are the same, nothing has changed about their data. What has changed is the definition of an attractive borrower, so concept drift has occurred in this case.
How does Oracle Machine Learning Services record model drift?
Fortunately, users of Oracle Machine Learning (OML) can now access model monitoring via the OML Services REST interface, specifically for classification and regression models.
So how does this new feature track a model for potential degradation? It does so by comparing the quality of predictions on a baseline dataset, usually the data the model was trained on, to the prediction quality on a new set of data. The difference in the quality of the model’s predictions on these two sets of data quantifies the drift. Model quality is measured with performance metrics which can be accuracy, balanced accuracy, AUC, F1, precision, or recall for classification models. For regression, it can be mean, mean squared error, mean absolute error, mean squared log error, or median absolute error. A user picks one of these metrics they would like to use, then drift is calculated as the difference between the value of the accuracy metric in the baseline period and its value in the new data period scaled by the value in the baseline period, which yields a result on a 0-1 scale. Drift can be calculated over multiple periods at once. For example, monitoring for model drift monthly over a period of multiple months. Prediction quality on each month of data would be compared independently against prediction quality on the the baseline data to identify potential drift. Additionally, data can be added dynamically to the new dataset, so as data grows over time new observations can be included in the monitoring.
On top of the model drift measurements, model monitoring provides further information about the model. This includes feature impact values for the most impactful features in the model, statistics that describe the distribution of predicted values, and histogram data to aid in visualization of the distribution of predictions. All this data, alongside the drift measurement, gives users a complete picture of how their model has drifted.
Walkthrough of model monitoring with OML Services
We are going to walk through an example of this new model monitoring feature in OML Services. We build a model on household electricity consumption data from 2007-2008, which is the same data we used in our data monitoring blog. We will then monitor the quality of this model as it makes predictions on new data from 2008-2010. To monitor a model, it must be deployed to OML Services. If you do not know how to deploy a model, check out the documentation on deploying a model from an AutoML experiment or the example notebook titled “OML4SQL Exporting Serialized Models,” which details how to deploy other models.
You access OML Services through a REST interface, and it is helpful to use an API platform, like Postman. We use Postman in this blog and it is available for download here. A collection of Postman requests for model monitoring can be found in our GitHub repository. If you prefer to use a terminal, a collection of cURL requests for model monitoring can be found in our model monitoring example notebook in the examples section of OML, or in our GitHub repository. We recommend you use these collections when you create your own requests, as using them will be much easier than building requests from scratch. Once you have gained familiarity with these requests, you will likely want to make these calls from code. Conveniently, Postman allows you to export your request in many common coding languages and frameworks.
Postman gives you the ability to store environment variables for use in your requests. When setting up the environment, store the following values before using the provided Postman requests:
- Omlserver is the base url of your OML instance. This is the url you use to access the notebook environment excluding the path.
- Username is the username used to log into your OML instance.
- Password is the password used to log into your OML instance.
These variables can be used in your Postman requests by placing the variable name in double brackets. You will note that throughout this blog, requests will include these bracketed variables. If you choose not to use variables, replace these with their actual values. If you are using a terminal instead of an API platform, it’s helpful to store these values as environment variables.
Step 1: Get Access Token
Before sending requests to create your model monitoring job, we need to get an access token from OML Services. To do so, we send a request to the /omlusers/api/oauth2/v1/token endpoint. The omlserver, username, and password variables are used here.
Note the response at the bottom of the image, which includes the access token (blurred in image). This token is another value that is convenient to save as an environment variable. In all requests that use this token for authentication, you must go to the Authentication tab, set the type as Bearer Token, and specify this as the token being used. If you are using the Postman collection provided for this blog, then all requests in that collection will automatically use this token for authentication.
Step 2: Create Model Monitoring Job
Now that we have an access token, we can send requests to OML Services. Let’s look at the data we are going to build our model on. This household electricity consumption dataset includes various consumption metrics of a household from 2006 to 2010. SUB_METERING_1 corresponds to the kitchen, SUB_METERING_2 to the laundry room, and SUB_METERING_3 to an electric water heater and air conditioner. We will build a regression model using data from 2006-2007 to predict the GLOBAL_ACTIVE_POWER variable, then monitor its performance as we make predictions on data from 2008-2010.
Now let’s create the model monitoring job request. This request will be sent as a POST request to the /omlmod/v1/jobs endpoint. There are many parameters that can be specified for a model monitoring job. Below is a listing of the required parameters. To learn about all the optional parameters, check out the documentation.
Required:
- jobName specifies the name of the job.
- jobType specifies the type of job to be run and is set to DATA_MONITORING for data monitoring jobs.
- outputData is the output data identifier. The results of the job will be written to a table named {jobId}_{ouputData}.
- baselineData is A table or view name that contains baseline data to monitor. At least 50 rows per period are required for monitoring, otherwise, analysis is skipped.
- newData is a table or view name with new data to be compared against the baseline. At least 50 rows per period are required for monitoring, otherwise, analysis is skipped.
- modelList is the list of models to be monitored, identified by their modelIds. The maximum length of this list is 20 models. If you do not know the modelID of your model, you can see a listing of all deployed models and their information by sending a GET request to the /omlmod/v1/models endpoint.
And here is what the request looks like:
Upon successfully submitting of your job request, you receive the following response. Take note of the jobId as you will need to use it to interact with the job, like viewing its details or updating it. We suggest you save it as an environment variable.
Step 3: View your job details
Once the job has been created, we can view its details by sending a GET request to the /omlmod/v1/jobs/{{jobId}}/ endpoint. This shows us the parameters we set in our job request, as well as information about the job’s last run if it ran at least once. By default, the job status is set to ENABLED. This means that the job will run according to the schedule we defined in the request.
Step 4: Update your job
Users have the option to update certain parameters of jobs after submitting a job request. For model monitoring jobs, the following parameters can be edited:
- startDate
- endDate
- threshold
- recompute
- modelList
We would like to change a few of the parameters of the request, so we will send a POST request to the /omlmod/v1/jobs/{{jobID}} endpoint. A successful request will get a 204 response with no body. This is what the request looks like:
Step 5: Perform a job action
Users can also perform actions on their jobs. The following actions can be performed on a model monitoring job:
- RUN will immediately run the job once. Good for testing job behavior.
- DISABLE will change the job state from ENABLE to DISABLE. This means that the job will no longer run according to its schedule but can still be run using a RUN action request.
- ENABLE will change the job state from DISABLE to ENABLE, allowing it to run according to its schedule again.
- STOP will stop a currently running job. An error will be returned if the job is not running.
These requests are sent as POST requests to the /omlmod/v1/jobs/{{jobID}}/action endpoint. Since we want to see the results of the model monitoring job without waiting for it to run according to its schedule, we send a request to this endpoint to run the job. A successful request will get a 204 response with no body.
Step 6: View model monitoring job output
We now know that the model monitoring job ran since we successfully sent the request to perform the RUN job action. If we wanted to ensure that the job ran successfully, we could check its details again as we did in step 3.
To finally view the model monitoring results, let’s return to OML Notebooks to run a database query and view the table holding the results. This table is called <jobId>_<outputData> where jobId is the id associated with the job and outputData is defined by the user in the job request.
In the first image, we see what the output table for model monitoring looks like. We can see the START_TIME and END_TIME telling us the period being monitored. The DRIFT, HAS_DRIFT, and THRESHOLD columns tell us that drift did occur in our model over this period. Our threshold was set to 0.10, and drift of 0.144 was observed. Additionally, the MODEL_METRICS column shows us many additional statistics to understand how our model has drifted. In the second image we visualize drift over the 3 years monitored. This shows us that essentially no drift occurred in 2008, a small amount occurred in 2009, then a significant amount of drift occurred in the final year monitored, 2010. For an in-depth explanation of monitoring results, check out the documentation.
Further Reading
After reading this blog and getting your hands on the new model monitoring feature, if you haven’t aready, check out my companion blog on data monitoring. As mentioned in this blog, data monitoring is a concept that compliments model monitoring in understanding how and why a model is degrading by learning about the data that it is making predictions on.
Here are some other useful links: