A model that works in a notebook is a promising result. A model that works in production is a system. The distance between the two is greater than it looks, and it is where most machine learning projects stall: the model is good enough, but it cannot be operated, repeated or trusted when the data changes.
This post collects the things that need to be in place before a model can carry part of a real system. As an example we use the fictitious company Space Travel, which wants to predict the demand for its trips in order to plan capacity.
The gap between notebook and production
In a notebook everything is under control: an extract of data that does not change, code that is run in a particular order, and a person looking at the result. In production none of that is true. Data arrives continuously and looks a little different every month, the code runs unattended, and the result is used by other systems that cannot tell whether it makes sense.
So the question is not only "how accurate is the model?", but also:
- Can we train exactly the same model again?
- Is the input computed the same way during training and in production?
- Do we notice when the model gets worse?
- What happens when the model cannot answer?
Reproducibility and data versioning
A model is the result of code, data, parameters and environment. Change one of them and you get a different model. To recreate a model, explain a prediction or roll back to the previous version, all four have to be findable again:
- Code in git, like all other code.
- Data versioned, so you know exactly which extract a model was trained on. That can be a tool such as DVC, a versioned data store or simply immutable, dated extracts with a checksum.
- Parameters and results recorded for every training run, so runs can be compared.
- Environment pinned: library versions, and preferably a container image, so training runs the same on any machine.
Every model that is released gets a version, and the version comes with every answer the model gives. Then you can always see which model was behind a particular prediction.
The same features in training and in production
One of the most common mistakes is that input is computed one way during training and another way in production, for example because training uses a prepared extract while production computes the same values live in a different language. The differences are often small and invisible, but the model notices them. The best protection is to have one implementation of the feature computation that both training and production use, and a test that compares the two on the same raw data.
Monitoring drift
A model does not get worse because the code changes. It gets worse because the world does. There are three things to watch:
- Data drift: the distribution of the input changes, for example because new customer groups arrive or a source changes its format.
- Prediction drift: the distribution of the model's answers changes, even though the input looks the same.
- Concept drift: the relationship between the input and the right answer changes. It can only be measured once the right answers are known, often weeks later.
Monitoring should not just show charts, but trigger action: an alert when a distribution has moved more than agreed, and a plan for who decides whether the model needs retraining.
Fallback strategies
Sooner or later a model will not be able to answer: the service is down, the input is invalid, or the answer is so uncertain that it should not be used. The system around the model has to know what to do then. Typical strategies are:
- A simple rule, for example last year's demand for the same trip and week.
- The previous model, if the new one behaves unexpectedly.
- A safe default and a flag that the answer did not come from the model.
The most important thing is that the fallback is a deliberate, tested choice, and that the response shows when it was used. Otherwise nobody notices that the model has not answered anything for a week.
Integration through an API
The safest way to connect a model to the rest of the system is to put it behind an API with a clear contract, like any other service: validated input, a timeout, versioned output and errors in a fixed format. The response should say which model answered, how confident it was, and whether a fallback was used:
{
"tripId": "MARS-2031-07",
"week": "2031-W27",
"predictedBookings": 184,
"interval": {
"low": 151,
"high": 219
},
"model": {
"name": "demand-forecast",
"version": "2031.06.2"
},
"fallback": false
}{
"tripId": "MARS-2031-07",
"week": "2031-W27",
"predictedBookings": 170,
"interval": null,
"model": null,
"fallback": true,
"fallbackReason": "model-timeout",
"fallbackRule": "same-week-last-year"
}With a contract like that, the rest of the system can treat the model as an ordinary dependency: it can set timeouts, log the version, fall back when needed, and switch models without changing the clients.
Release carefully
A new model should not go straight from testing to all users. Two simple techniques reduce the risk:
- Shadow mode: the new model gets the same requests as the old one, but its answers are not used. You just compare.
- Gradual rollout: the new model first answers a small share of the requests, and the share grows once the numbers look sensible.
Both require the model version to be logged with every answer, and switching back to be easy. It is the same discipline as for any other release; machine learning just makes it more necessary.
In short
- A model in production is a system: code, data, parameters and environment must be recreatable.
- Version the data, and let the model version come with every answer.
- Compute features in one place, so training and production see the same input.
- Monitor data, prediction and concept drift, and tie alerts to a plan.
- Have a tested fallback, and show in the response when it is used.
- Put the model behind an API with a clear contract, and release new models in shadow mode or gradually.
Does your model need to go from notebook to production?
We are happy to help with reproducible training, integration through an API, monitoring and fallbacks, so the model can carry part of a real system. Let us have an informal chat.
- machine-learning
- mlops
- drift
