Can Machine Learning Systems Heal Themselves? Building a Self-Healing ML System — Phase 1
Designing an automated framework for detecting machine learning failures, diagnosing their causes, and evaluating recovery strategies. Series: Building a Self-Healing Machine Learning System | Part 1 Introduction: What Happens When Machine Learning Models Fail? Machine learning models are often eva
Posted on Oct 24 • Originally published at Medium
Designing an automated framework for detecting machine learning failures, diagnosing their causes, and evaluating recovery strategies. Series: Building a Self-Healing Machine Learning System | Part 1 Introduction: What Happens When Machine Learning Models Fail? Machine learning models are often evaluated under carefully controlled conditions.
We train models, measure performance, optimize hyperparameters, and deploy the best-performing version. But what happens after deployment? Real-world data rarely remains unchanged.
Customer behavior evolves. Data pipelines introduce unexpected missing values. Feature distributions shift.
Previously reliable predictions can gradually become less accurate. A model that performs well during development may behave very differently when exposed to changing production data. Traditional machine learning workflows often focus on detecting performance degradation through monitoring dashboards, periodic evaluations, and manual retraining.
But I wanted to explore a different question: Can a machine learning system automatically detect a failure, identify its likely cause, apply an appropriate recovery strategy, and determine whether the recovered model is safe to use? That question became the foundation of my graduate capstone research project: Self-Healing Machine Learning System for Automated Failure Detection and Model Recovery in Credit-Default Prediction. In this first article, I will walk through Phase 1 of the project, where I defined the research problem, selected the dataset, designed the system architecture, established evaluation criteria, and developed a reproducible experimental plan.
Rather than immediately implementing a complex machine learning model, I began by defining what failure, recovery, and successful healing should actually mean.
I chose this dataset for several reasons. First, credit-default prediction represents a meaningful application where model reliability matters. Incorrect predictions can have consequences for financial decisions.
Second, the dataset contains both numerical and categorical features, making it suitable for experimenting with multiple types of data-quality and distribution failures. Third, its manageable size allows me to run controlled experiments, reproduce results, and compare recovery strategies without requiring extensive computing infrastructure. Finally, the dataset provides a useful environment for separating three research questions: Can the system detect a problem?
Can it correct the problem? And does the correction preserve predictive performance? Although the dataset provides a realistic classification problem, my experiments use historical data and simulated failures.
They do not constitute a production credit-decision system.
My baseline pipeline uses StandardScaler for numerical feature scaling and Logistic Regression for classification. I also planned a small TensorFlow/Keras multilayer perceptron as a future comparison model. This will eventually allow me to investigate whether different model architectures respond differently to identical data failures.
These controlled scenarios allow me to systematically investigate how different levels of missingness in the BILL_AMT1 feature influence model performance, failure detection, and the effectiveness of the recovery strategy. These scenarios provide a controlled way to examine how increasing missingness affects classifier performance. Rather than changing multiple features simultaneously, I initially focus on one feature so that the failure mechanism remains interpretable. This design makes it easier to compare outcomes and isolate the effect of the injected corruption. The experiment is conducted on held-out validation data, while training-derived statistics remain fixed. This separation is important for preventing information leakage.
The planned data split is: 70% training, 15% validation, and 15% testing. Validation data supports controlled failure experiments and recovery assessment, while the held-out test set provides a separate resource for final evaluation. I also intend to document unsuccessful recovery attempts rather than reporting only favorable results. A rejected recovery is not necessarily a failed research experiment. It can provide important evidence about the limitations of the correction strategy.