New perspectives on an ever-evolving role

Image by Author

Just fifty years ago, machine learning was a new idea. Today it’s an integral part of society, helping people do everything from driving cars and finding jobs to getting loans and receiving novel medical treatments.

When we think about what the next 50 years of ML will look like, it’s impossible to predict. New, unforeseen advancements in everything from chips and infrastructure to data sources and model observability have the power to change the trajectory of the industry almost overnight.

That said, we know that the long run is just a collection of short runs, and in the current run…

On defining, measuring, and fixing drift in your machine learning models

This blog was written in collaboration with Hua Ai, Data Science Manager at Delta Air Lines. In this piece, Hua and Aparna Dhinakaran, CPO and co-founder of Arize AI, discuss how to monitor and troubleshoot model drift.

Image by Author

As an ML practitioner, you probably have heard of drift. In this piece, we will dive into what drift is, why it’s important to keep track of, and how to troubleshoot and resolve the underlying issue when drift occurs.

What is Drift

First things first, what is drift? Drift is a change in distribution over time. It can be measured for model inputs, outputs, and actuals…

Notes from Industry

Building machine learning proof of concepts in the lab is drastically different from making models that work in the real world.

As more and more teams turn to machine learning to streamline their businesses or turn previously impractical technologies into reality, there has been a rising interest in tools and people who can help bring a model from the research lab into the customer’s hands. Google built TFX, Facebook built FBLearner, Uber built Michaelangelo, Airbnb built Bighead, and these systems have allowed these times to scale their MLOps.

Outside of these large tech companies, the truth is, building machine learning proof of concepts in the lab is drastically different from making models that work in the real world. …

Written in collaboration with Barr Moses, CEO, Monte Carlo

As data and machine learning ecosystems become increasingly complex and companies ingest more and more data, it’s important that data and ML engineering teams go beyond monitoring to understand the health of their data-driven systems.

One duplicate data set or stale model can cause unintended (but severe) consequences that monitoring alone can’t catch or prevent. The solution? Observability. Here’s how it differs from traditional monitoring and why it’s necessary for building more trustworthy and reliable systems.

Garbage in, garbage out

It’s a common saying among data and ML teams for good reason — but in…

Image by Author

At a rapid pace, many machine learning techniques have moved from proof of concepts to powering crucial pieces of technology that people rely on daily. In attempts to capture this newly unlocked value, many teams have found themselves caught up in the fervor of productionizing machine learning in their product without the right tools to do so successfully.

The truth is, we are in the early innings of defining what the right tooling suite will look like for building, deploying, and iterating on machine learning models. …

ML OBSERVABILITY SERIES

Performance Monitoring of ML Models

As Machine Learning infrastructure has matured, the need for model monitoring has surged. Unfortunately this growing demand has not led to a foolproof playbook that explains to teams how to measure their model’s performance.

Performance analysis of production models can be complex, and every situation comes with its own set of challenges. Unfortunately, not every model application scenario has an obvious path to measuring performance like the toy problems that are taught in school.

In this piece we will cover a number of challenges connected to availability of ground truth and discuss the performance metrics that are available to measure…

ML Observability Series

Part 2: Model Failure Modes

Staples Button, But Make it Machine Learning, Image by Author

In our last post we took a broad look at model observability and the role it serves in the machine learning workflow. In particular, we discussed the promise of model observability & model monitoring tools in detecting, diagnosing, and explaining regressions models that have been deployed to production.

This leads us to a natural question of: what should I monitor in production? The answer, of course, depends on what can go wrong.

In this article we will be providing some more concrete examples of potential failure modes along with the most common symptoms that they exhibit in your production model’s…

ML OBSERVABILITY SERIES

Part 1: The ML Observability Series

ML Observability Platform Image by Author

Machine Learning (ML) is being adopted by businesses in almost every industry. Many businesses are looking towards ML Infrastructure platforms to propel their movement of leveraging AI in their business. Understanding the various platforms and offerings can be a challenge. The ML Infrastructure space is crowded, confusing, and complex.

To understand the ecosystem, we broadly segment the machine learning workflow into three stages — data preparation, model building, and production. …

Image by Author. Statistical Checks: Inputs, Outputs and Actuals

Introduction

Statistical Distances are used to quantify the distance between two distributions and are extremely useful in ML observability. This blog post will go into statistical distance measures and how they are used to detect common machine learning model failure modes.

Why Statistical Distance Checks?

Data problems in Machine Learning can come in a wide variety that range from sudden data pipeline failures to long-term drift in feature inputs. Statistical distance measures give teams an indication of changes in the data affecting a model and insights for troubleshooting. …

Part 2 — Model Deployment and Serving

Businesses in almost every industry are adopting Machine Learning (ML) technology. Businesses look towards ML Infrastructure platforms to help them best leverage artificial intelligence (AI).

Understanding the various platforms and offerings can be a challenge. The ML Infrastructure space is crowded, confusing, and complex. There are a number of platforms and tools, which each have a variety of functions across the model building workflow.

To understand the ML infrastructure ecosystem, we can broadly segment the machine learning workflow into three stages — data preparation, model building, and production. …

Aparna Dhinakaran

Co-Founder and CPO of Arize AI. Formerly Computer Vision PhD at Cornell, Uber Machine Learning, UC Berkeley AI Research.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store