Industrial MLOps: Drift Monitoring and Human-in-the-Loop

An industrial AI model does not fail on the day it is deployed: it fails six months later, when the raw material changes, a bearing is replaced or a new product comes in and nobody is watching. Industrial MLOps is the discipline that keeps models alive on the shop floor: monitoring data and concept drift, setting confidence thresholds so the model knows when to abstain, feeding the operator's corrections into the learning cycle, and deciding with clear criteria when to retrain and when to retire a model.

Why industrial models stop working

Every machine learning model is a photograph. It learns the relationship between some inputs (vibration, temperature, images of the part, consumption) and an output (imminent failure, defective part, forecast demand) exactly as that relationship existed in the training data. The problem is that a plant is not a photograph: it is a film. Suppliers change, machines wear, tooling is replaced, recipes are adjusted and the seasons alter the temperature of the workshop. Each of those changes pulls reality a little further away from the photograph the model learned.

This degradation is silent, and that is where the danger lies. A broken bearing can be heard; a degraded model cannot. The system keeps returning predictions with the same air of confidence as on day one, only now they are worse. If nobody measures the quality of those predictions continuously, the first sign that the model has died is usually a batch rejected by the customer or a breakdown the predictive system never flagged. That is why serious work with industrial AI does not end at deployment: it starts there. The advanced AI models that are fit for production are the ones that arrive with their surveillance system attached, not the ones that score best in the lab.

To ground the ideas we will use one example throughout the article: a plastic injection line with a vision model classifying parts as good or defective at the mould exit. The model was trained on six months of production images and, in validation, comfortably outperformed the human inspector in consistency. We will see how that model degrades, how to detect it, and what to do at each stage.

Data drift and concept drift, with shop floor examples

The literature distinguishes two main forms of degradation, and the distinction matters because they are detected and corrected differently.

Data drift: what comes in changes

Data drift (or covariate shift) occurs when the distribution of the inputs moves away from what the model saw in training, even though the relationship between input and output stays the same. On our injection line: the purchasing department switches pellet supplier. The new polymer meets the same datasheet, but its base tone is slightly more opaque. The parts are still good or bad for the same reasons as always, but every image reaching the model is now a step outside the training distribution. The model starts flagging perfectly valid parts as suspect, because their surface no longer looks like anything it learned.

Other common shop floor examples: a camera replaced by a different model with different optics, a vibration sensor repositioned after maintenance, a new LED light fitting above the inspection station, or a shift change that moves machine parameters within tolerances. None of this is anyone's fault: it is the normal life of a factory.

Concept drift: what it means changes

Concept drift runs deeper: the very relationship between inputs and output changes. What was a defect yesterday is not one today, or the other way round. Continuing the example: the main customer tightens its cosmetic criterion and a surface streak that used to be accepted becomes grounds for rejection. The images have not changed at all; the definition of a bad part has. The model may be seeing exactly what it saw in training and still be wrong about every streaked part, because it learned a rule that no longer applies.

In predictive maintenance, concept drift appears when a machine or a critical component is replaced: the vibration pattern that anticipated the old bearing's failure anticipates nothing in the new bearing, which will fail differently and with different spectral signatures. A change of operating regime causes it too: moving from long production runs to short ones alters what a normal start-up means.

How to detect drift: metrics and monitoring

Detection is organised in three layers, from the cheapest to the most reliable.

First layer: watch the inputs. You do not need to know whether the prediction was correct to notice that the data has changed. The distributions of the input variables (or of the embeddings, in vision) are compared continuously against a reference window from training. Common statistical tools: the Kolmogorov-Smirnov test for continuous variables, the population stability index (PSI) that banking has used for decades on its risk models, or distances between distributions such as Wasserstein. This layer detects data drift almost in real time and is the first sensible alarm.

Second layer: watch the model's outputs. Even without knowing the truth yet, the distribution of predictions itself is informative. If the part classifier went from flagging a stable percentage of rejects to doubling it in two days with no known change in the process, something is going on: either the process has genuinely degraded or the model is disoriented. Both deserve a visit to the line. A drop in the average confidence of predictions is another useful signal at this layer.

Third layer: compare against reality. It is the only one that detects concept drift, and it requires labels: knowing, for a sample of predictions, what actually happened. In visual inspection, periodic sampling reviewed by a human inspector. In predictive maintenance, the contrast between alarms raised and actual breakdowns logged in the CMMS. This layer is slower (the truth takes time to arrive) but it measures what matters: precision, recall, false positives per shift. From deployment onwards, define which metric is critical and which value triggers a review, exactly as you would with any other process indicator.

These three layers are not only MLOps theory: they are the same mechanics that underpin the Captia.ai anomaly detection module, which compares each signal against the asset historical normality and surfaces the deviation as an event and an alert inside the platform. That page details what data it consumes, how sensitivity is tuned and when the module does not apply.

Confidence thresholds: deciding when the model does not decide

A well-operated model does not always answer. Most classification models return, alongside the prediction, a confidence score. The operational design consists of turning that score into three decision zones:

1. High confidence in a good part: the part passes without human intervention. This is the zone that generates the savings.
2. High confidence in a bad part: automatic rejection, with the part set aside for verification by sampling.
3. Intermediate zone: the model abstains and the part goes to human review. Here the system acknowledges that it does not know.

The thresholds separating these zones are not a technical parameter: they are a business decision. Raising the automatic zone's threshold reduces the risk of letting a defect through, but sends more parts to review and erodes the savings. The balance point depends on the cost of each type of error: on a low-value cosmetic part you can tolerate lax thresholds; on a safety component, the abstention zone must be generous. And there is a valuable side effect: the size of the intermediate zone is itself a drift detector. If the percentage of parts on which the model abstains goes from the usual 5% to 20%, the model is shouting that the world has changed, even before any accuracy metric confirms it.

Human-in-the-loop: the operator corrects and the system learns

The abstention zone has a second function, more important than the first: it manufactures training data. Every time the operator reviews a doubtful part and pronounces the verdict, they are labelling exactly the cases where the model is weakest. A well-designed human-in-the-loop system captures that verdict in a structured way (not in a notebook, nor in the inspector's head) and incorporates it into the dataset for the next retraining. The result is a self-reinforcing cycle: the model routes the hard cases to the human, the human resolves them, and those resolved cases mean the next model version has fewer hard cases.

For the cycle to work on the shop floor there are three practical conditions. First: the correction must cost seconds, not minutes. If validating a part means opening a separate application and filling in four fields, the operator will stop doing it by day three, and rightly so. A physical button, or two taps on the station's screen. Second: the operator has to see that their corrections count. When the new model version stops making the mistake they corrected forty times, the system earns their trust; if their corrections vanish into a pit, it earns their contempt. Third: the corrections are audited too. Human inspectors disagree with each other (any attribute repeatability and reproducibility study confirms it), so contradictory human labels must be detected and arbitrated before training on them, or the model will learn the inconsistency.

This collaboration pattern is the same one that underpins AI agents in operations: the system acts alone within its proven zone of competence, escalates to the human whatever falls outside it, and the border between the two zones moves with evidence, not with enthusiasm.

Retraining and validation before returning to production

With drift detected and corrections accumulated, it is time to retrain. Two symmetrical mistakes are worth avoiding. The first is retraining hot and blind: pouring in the new data, training and deploying without further ado. A retraining can make the model worse (badly labelled new data, overfitting to a transient episode) and without validation there is no way to know until the damage is done. The second mistake is never retraining for fear of touching what works, which is like never changing the oil so as not to open the engine.

The disciplined process has four steps:

1. Freeze an honest validation set. Labelled parts from the last few weeks, reflecting current reality (with the new pellets, with the new criterion), kept separate from training. Old model and new model are compared against that set with the same metrics.
2. Train as a candidate, not as a successor. The new version does not inherit the throne for being new: it has to beat the current one on the validation set and not degrade on the cases the current one already solved well.
3. Deploy in shadow mode. Before handing over control, the candidate runs in parallel on real production without acting: it predicts, it is logged, and its predictions are compared with those of the active model and with the human verdicts for days or weeks depending on the risk of the process.
4. Promote with a rollback ready. Every version is recorded with its data, its code and its metrics (model versioning practices and the registries common in MLOps help here), so that returning to the previous version is a matter of minutes, not a project.

The cadence depends on the process: there are plants where a scheduled quarterly retraining is enough, and processes with seasonal drift or frequent raw material changes where the trigger must be the monitoring metric, not the calendar. What matters is that a defined trigger exists, whichever it is.

When to retire a model (and how to do it calmly)

Not every model deserves rescuing. There are three situations where the right decision is retirement, and it pays to recognise them early, because a degraded model that keeps operating destroys two things at once: quality, and the workforce's trust in AI.

• The problem no longer exists. The machine the predictive model watched has been replaced, the product it inspected has been discontinued. Keeping the model running is pure cost and meaningless alarm noise.
• Retraining no longer pays. If every few months thousands of samples have to be relabelled to chase a process that changes faster than the model learns, the maintenance cost exceeds the benefit. Sometimes the right answer is a different approach (other variables, another type of model, or a physical process improvement that reduces the variability at source).
• Operational trust is broken. If operators have learned to ignore the alarms because they fail too often, the model is already retired de facto; formalising it is more honest than pretending it is still in service.

An orderly retirement replicates the deployment in reverse: the process is explicitly and openly handed back to the manual procedure or the previous system, the final version is archived with its data and metrics (the history is gold for the next attempt) and the reason for retirement is documented. A retired model with its post-mortem written up is an asset; a model abandoned in production is a liability.

The minimum MLOps architecture for a plant

None of the above requires a pharaonic platform. For an industrial SME with one or two models in production, the minimum viable architecture comes down to five pieces:

Note that three of the five pieces are organisational rather than technological. Industrial MLOps looks less like buying a platform and more like extending to models the same discipline the plant already applies to its instrument calibrations: reference standard, periodic verification, records and an intervention criterion. Whoever runs their weighing scales well already knows how to run their models; they just need to transfer the habit.

Frequently asked questions about industrial MLOps

If you have models in production without drift monitoring, or AI projects that never made it past the pilot because nobody defined how to maintain them, at Captia AI we support that whole cycle: from deployment to continuous operation, with the operator inside the loop from day one.