Published in Towards Data Science·Mar 2How to Understand and Use the Jensen-Shannon DivergenceA primer on the math, logic, and pragmatic application of JS Divergence — including how it is best used in drift monitoring — This piece is co-authored with Jason Lopatecki, CEO and Co-Founder of Arize AI In machine learning systems, drift monitoring can be critical to delivering quality ML. Some common use cases for drift analysis in production ML systems include: Detect feature changes between training and production to catch problems ahead of…Data Drift9 min readData Drift9 min read
Published in Towards Data Science·Feb 2Understanding KL DivergenceA guide to the math, intuition, and practical use of KL divergence — including how it is best used in drift monitoring — Kullback-Leibler divergence metric (relative entropy) is a statistical measurement from information theory that is commonly used to quantify the difference between one probability distribution from a reference probability distribution. While it is popular, KL divergence is sometimes misunderstood. …Model Drift7 min readModel Drift7 min read
Published in Towards Data Science·Jan 25Demystifying NDCGHow to best use this important metric for monitoring ranking models — This piece is co-authored by Jianshu Chi, Software Engineer at Arize AI, and Amber Roberts, ML engineer at Arize AI Ranking models underpin many aspects of modern digital life, from search results to music recommendations. …Ndcg11 min readNdcg11 min read
Published in Towards Data Science·Jan 19A Quickstart Guide To Uprooting Model BiasThis article is co-authored by Amber Roberts, Machine Learning Engineer at Arize AI In today’s world, it is all too common to read about AI acting in discriminatory ways. From real estate valuation models that reflect the continued legacy of housing discrimination to models used in healthcare that amplify inequities…Fairness And Bias12 min readFairness And Bias12 min read
Published in Towards Data Science·Dec 8, 2022Measuring Embedding DriftApproaches for measuring embedding/vector drift for unstructured data, including for computer vision and natural language processing models — This article was written in collaboration with Jason Lopatecki, CEO and Co-Founder of Arize AI Data drift in unstructured data like images is complicated to measure. The metrics typically used for drift in structured data — such as population stability index (PSI), Kullback-Leibler divergence (KL divergence), and Jensen-Shannon divergence (JS…NLP9 min readNLP9 min read
Published in Towards Data Science·Jul 20, 2022Three Pitfalls To Avoid With EmbeddingsWritten in collaboration with Francisco Castillo Carrasco, data scientist at Arize AI. Introduction Let’s say that you have read a very helpful post demystifying embeddings and you’re really excited. Your social media company can certainly use them, so you fire up your notebook and start typing away. …Embedding6 min readEmbedding6 min read
Published in Towards Data Science·Jun 15, 2022The Three Types of Observability Your System NeedsThis article is written in partnership with Kyle Kirwan, Co-founder and CEO at Bigeye What Is Observability? In 1969, humans first stepped on the moon thanks to a lot of clever engineering and 150,000 lines of code. Among other things, this code enabled engineers at mission control to have the full view of…Observability8 min readObservability8 min read
Published in Towards Data Science·Jun 2, 2022Getting Started With Embeddings Is Easier Than You ThinkA quick guide to understanding embeddings, including real world applications and how to compute them — Written in collaboration with Francisco Castillo Carrasco, data scientist at Arize AI. Imagine you are an engineer at a promising chatbot startup dedicated to helping people find the medical care they need quickly. You have millions of chat interactions between customers and medical personnel. You’re building a model to route…Embedding8 min readEmbedding8 min read
Published in Towards Data Science·Mar 29, 2022Introducing ML Performance TracingPart Two In a Series On ML Troubleshooting — In part one of this content series, we covered how painful ML troubleshooting is today and how to get started with ML monitoring. However, monitoring alone does NOT lead to resolution. To illustrate this, let’s revisit the situation from part one of this series. You are enjoying your morning coffee…Performance Tracing10 min readPerformance Tracing10 min read
Published in Towards Data Science·Feb 25, 2022ML Troubleshooting Is Too Hard Today (But It Doesn’t Have to Be That Way)As ML practitioners deploy more models into production, the stakes for model performance are higher than ever — and the mistakes costlier. — Part One: From No Monitoring to Monitoring To paraphrase a common bit of wisdom, if a machine learning model runs in production and no one is complaining, does it mean the model is perfect? The unfortunate truth is that production models are usually left alone unless they lead to negative business impacts.Model Monitoring9 min readModel Monitoring9 min read
