Published inTowards Data Science·May 23How To Best Leverage OpenAI’s Evals FrameworkKeys for evaluating LLMs using OpenAI Evals — This blog is co-authored by Trevor LaViale According to a recent survey, nearly half (43%) of data science and engineering teams are planning production deployments of large language models (LLMs) over the next year. The age of LLMs is definitely upon us; however, evaluating these models is often challenging, and…OpenAI6 min readOpenAI6 min read
Published inTowards Data Science·Apr 25Applying Large Language Models to Tabular Data to Identify DriftCan LLMs reduce the effort involved in anomaly detection, sidestepping the need for parameterization or dedicated model training? — Follow along with this blog’s accompanying colab. This blog is a collaboration with Jason Lopatecki, CEO and Co-Founder of Arize AI, and Christopher Brown, CEO and Founder of Decision Patterns Recent advances in large language models (LLM) are proving to be a disruptive force in many fields (see: Sparks of…Large Language Models9 min readLarge Language Models9 min read
Published inTowards Data Science·Apr 6Boosting Tabular Data Predictions with Large Language ModelsWhat happens when you unleash GPT-4 on a tabular Kaggle competition to predict home prices? — Follow along with this blog’s accompanying Colab. This blog is a collaboration with Jason Lopatecki, CEO and Co-Founder of Arize AI, and Christopher Brown, CEO and Founder of Decision Patterns Introduction There are two distinct groups in the ML ecosystem. One works with highly organized data collected in tables — the…Large Language Models9 min readLarge Language Models9 min read
Published inTowards Data Science·Mar 2How to Understand and Use the Jensen-Shannon DivergenceA primer on the math, logic, and pragmatic application of JS Divergence — including how it is best used in drift monitoring — This piece is co-authored with Jason Lopatecki, CEO and Co-Founder of Arize AI In machine learning systems, drift monitoring can be critical to delivering quality ML. Some common use cases for drift analysis in production ML systems include: Detect feature changes between training and production to catch problems ahead of…Data Drift9 min readData Drift9 min read
Published inTowards Data Science·Feb 2Understanding KL DivergenceA guide to the math, intuition, and practical use of KL divergence — including how it is best used in drift monitoring — Kullback-Leibler divergence metric (relative entropy) is a statistical measurement from information theory that is commonly used to quantify the difference between one probability distribution from a reference probability distribution. While it is popular, KL divergence is sometimes misunderstood. In practice, it can also sometimes be difficult to know when to…Model Drift7 min readModel Drift7 min read
Published inTowards Data Science·Jan 25Demystifying NDCGHow to best use this important metric for monitoring ranking models — This piece is co-authored by Jianshu Chi, Software Engineer at Arize AI, and Amber Roberts, ML engineer at Arize AI Ranking models underpin many aspects of modern digital life, from search results to music recommendations. Anyone who has built a recommendation system understands the many challenges that come from developing…Ndcg11 min readNdcg11 min read
Published inTowards Data Science·Jan 19A Quickstart Guide To Uprooting Model BiasThis article is co-authored by Amber Roberts, Machine Learning Engineer at Arize AI In today’s world, it is all too common to read about AI acting in discriminatory ways. From real estate valuation models that reflect the continued legacy of housing discrimination to models used in healthcare that amplify inequities…Fairness And Bias12 min readFairness And Bias12 min read
Published inTowards Data Science·Dec 8, 2022Measuring Embedding DriftApproaches for measuring embedding/vector drift for unstructured data, including for computer vision and natural language processing models — This article was written in collaboration with Jason Lopatecki, CEO and Co-Founder of Arize AI Data drift in unstructured data like images is complicated to measure. The metrics typically used for drift in structured data — such as population stability index (PSI), Kullback-Leibler divergence (KL divergence), and Jensen-Shannon divergence (JS…NLP9 min readNLP9 min read
Published inTowards Data Science·Jul 20, 2022Three Pitfalls To Avoid With EmbeddingsWritten in collaboration with Francisco Castillo Carrasco, data scientist at Arize AI. Introduction Let’s say that you have read a very helpful post demystifying embeddings and you’re really excited. Your social media company can certainly use them, so you fire up your notebook and start typing away. …Embedding6 min readEmbedding6 min read
Published inTowards Data Science·Jun 15, 2022The Three Types of Observability Your System NeedsThis article is written in partnership with Kyle Kirwan, Co-founder and CEO at Bigeye What Is Observability? In 1969, humans first stepped on the moon thanks to a lot of clever engineering and 150,000 lines of code. Among other things, this code enabled engineers at mission control to have the full view of…Observability8 min readObservability8 min read
