Project Overview
A US technology startup had built an ML model internally to power a core feature of their product. The model performed well in testing, but when it was deployed to production, it produced inconsistent and unreliable results that users could observe. The problem had persisted for months without a diagnosis. The team had ruled out several possible causes on their own but had no MLOps experience and lacked the tooling to investigate systematically. We were brought in to diagnose the root cause and implement a durable fix.
The Challenge
The startup was in a difficult position: they had a model they knew could work, evidence from testing proved it, but they could not replicate that performance in production. Without MLOps infrastructure or prior experience debugging serving pipelines, the team had no structured way to narrow down the cause.
- ML model producing inconsistent and unreliable results in production despite strong testing performance
- Problem had been present for months, long enough that multiple possible causes had accumulated and could not be easily ruled out
- No MLOps tooling or monitoring in place, so the team had no visibility into what the model was receiving as input in production or how predictions were distributed
- No systematic diagnostic process had been applied: the team was making individual changes and observing results without a controlled framework
- No data quality checks between the training pipeline and the serving pipeline
The combination of a months-long problem window and no observability infrastructure meant the first step had to be establishing the ability to see what was actually happening before any fix could be responsibly scoped.
Our Solution
We applied a systematic diagnostic process before writing a single line of fix code. The goal was to rule out causes in order of likelihood rather than attempt fixes in parallel, which would have made it impossible to isolate the actual cause.
- Pipeline Diagnostic Audit: We compared the full data flow from raw input through to model prediction at both the training stage and the serving stage, looking specifically for any point where the data structure, schema, or preprocessing logic diverged between the two environments.
- Root Cause Identification: We identified a data pipeline inconsistency creating training and serving skew. The model had been trained on data that had been preprocessed with one set of transformations, but the production serving pipeline was applying a different version of those transformations, feeding the model differently structured data than it had learned from. The model was not failing: it was working correctly on the wrong input.
- Pipeline Rebuild: We rebuilt the deployment pipeline to ensure consistent data preprocessing at both the training and serving stages, using a shared preprocessing module rather than two separately maintained implementations.
- MLOps Monitoring Implementation: We set up monitoring to detect data drift and data quality issues going forward, with alerts configured to flag when the distribution of production inputs deviated meaningfully from the training distribution, giving the team early warning before the problem could recur.
Technical Approach
The training and serving skew was caused by the preprocessing logic having been written twice: once in the training notebook and once in the serving API, with slight differences that had accumulated over time as one was updated and the other was not. The fix was architectural, consolidating preprocessing into a single shared library that both the training pipeline and the serving layer imported, eliminating the possibility of the two diverging in the future.
The MLOps monitoring layer was implemented to track input feature distributions and prediction output distributions in production, comparing them against reference distributions captured at training time. Alerts were connected to the team’s existing notification channel so any significant drift would surface immediately rather than being discovered through user complaints.
Results & Impact
- Production model performance matched testing benchmarks within two weeks of the pipeline fix going live
- Root cause identified and resolved: months of inconsistent production results were traced to a single architectural issue in the serving pipeline
- MLOps monitoring infrastructure deployed and active, with drift detection in place to catch similar data quality issues before they affect users in the future
- The team had a structured understanding of training and serving consistency requirements they had not had before, applicable to any future models they build
Lessons Learned
Training and serving skew is one of the most common and most underdiagnosed causes of ML model problems in production. It is particularly difficult to catch when teams are working quickly, because the natural tendency is to write the preprocessing logic once for training and then rewrite it for the serving context, treating them as two separate tasks rather than one shared component. The fix is not complicated, but identifying it requires systematic comparison of the full data path at both stages, which teams without MLOps experience rarely do by default.
The monitoring setup was as important as the pipeline fix. Without it, the same class of problem could have recurred the next time someone updated the serving logic without updating the training pipeline. Giving the team visibility into input and output distributions in production created a feedback loop that the startup had never had before, making future issues detectable before they reached users.