Monitor The Health Of Your Machine Learning Products In Production With Evidently
Summary
You’ve got a machine learning model trained and running in production, but that’s only half of the battle. Are you certain that it is still serving the predictions that you tested? Are the inputs within the range of tolerance that you designed? Monitoring machine learning products is an essential step of the story so that you know when it needs to be retrained against new data, or parameters need to be adjusted. In this episode Emeli Dral shares the work that she and her team at Evidently are doing to build an open source system for tracking and alerting on the health of your ML products in production. She discusses the ways that model drift can occur, the types of metrics that you need to track, and what to do when the health of your system is suffering. This is an important and complex aspect of the machine learning lifecycle, so give it a listen and then try out Evidently for your own projects.
Announcements
- Hello and welcome to Podcast.__init__, the podcast about Python’s role in data and science.
- When you’re ready to launch your next app or want to try a project you hear about on the show, you’ll need somewhere to deploy it, so take a look at our friends over at Linode. With the launch of their managed Kubernetes platform it’s easy to get started with the next generation of deployment and scaling, powered by the battle tested Linode platform, including simple pricing, node balancers, 40Gbit networking, dedicated CPU and GPU instances, and worldwide data centers. Go to pythonpodcast.com/linode and get a $100 credit to try out a Kubernetes cluster of your own. And don’t forget to thank them for their continued support of this show!
- Your host as usual is Tobias Macey and today I’m interviewing Emeli Dral about monitoring machine learning models in production with Evidently
Interview
- Introductions
- How did you get introduced to Python?
- Can you describe what Evidently is and the story behind it?
- What are the metrics that are useful for determining the performance and health of a machine learning model?
- What are the questions that you are trying to answer with those metrics?
- How does monitoring of machine learning models compare to monitoring of infrastructure or "traditional" software projects?
- What are the failure modes for a model?
- Can you describe the design and implementation of Evidently?
- How has the architecture changed or evolved since you started working on it?
- What categories of model is Evidently designed to work with?
- What are some strategies for making models conducive to monitoring?
- What is involved in monitoring a model on a continuous basis?
- What are some considerations when establishing useful thresholds for metrics to alert on?
- Once an alert has been triggered what is the process for resolving it?
- If the training process takes a long time, how can you mitigate the impact of a model failure until the new/updated version is deployed?
- What are the most interesting, innovative, or unexpected ways that you have seen Evidently used?
- What are the most interesting, unexpected, or challenging lessons that you have learned while working on Evidently?
- When is Evidently the wrong choice?
- What do you have planned for the future of Evidently?
Keep In Touch
- @EmeliDral on Twitter
- emeli-dral on GitHub
Picks
- Tobias
- Emeli
Links
The intro and outro music is from Requiem for a Fish The Freak Fandango Orchestra / CC BY-SA