Detecting Data Drift & Concept Drift in Production ML Systems

The Silent Failure Mode of Machine Learning

Unlike an API that returns HTTP 500 when a database connection dies, a machine learning model will happily accept drifted input distributions and output confident nonsense. If consumer behavior shifts, if a hospital purchases new fundus imaging cameras with slightly different color saturation, or if macroeconomic factors shift stock volatility, your model's accuracy deteriorates while CPU and latency metrics remain green.

Calculating Population Stability Index (PSI) in Python

The Population Stability Index (PSI) is an industry-standard metric for measuring divergence between a reference (baseline/training) distribution and an active production distribution:

import numpy as np

def calculate_psi(baseline: np.ndarray, current: np.ndarray, num_bins: int = 10) -> float:
    # Generate quantile bins from baseline distribution
    percentiles = np.linspace(0, 100, num_bins + 1)
    bin_edges = np.percentile(baseline, percentiles)
    bin_edges[0] -= 1e-5
    bin_edges[-1] += 1e-5

    baseline_counts, _ = np.histogram(baseline, bins=bin_edges)
    current_counts, _ = np.histogram(current, bins=bin_edges)

    baseline_pct = np.clip(baseline_counts / len(baseline), 1e-4, 1.0)
    current_pct = np.clip(current_counts / len(current), 1e-4, 1.0)

    # PSI Formula
    psi_value = np.sum((current_pct - baseline_pct) * np.log(current_pct / baseline_pct))
    return float(psi_value)

# Benchmarks:
# PSI < 0.10: Stable
# 0.10 <= PSI < 0.25: Moderate drift
# PSI >= 0.25: Severe drift