Monitoring models in production
A model is only finished when you can tell it is still working. The few checks I set up before every launch.
Training accuracy tells you very little about how a model behaves next month.
Four checks I add before launch
- Input drift. Compare live feature distributions with the training data.
- Score distribution. A sudden shift in predictions is often the first warning sign.
- Latency and errors. Treat the model like any other service.
- Business outcome. Track the metric the model exists to improve, not only a proxy.
Keep alerts few and meaningful
An alert that fires every day gets ignored. Start with a small set, tune the thresholds, and review them after each incident.