MLOps CI/CD Consulting: How We Build ML Pipelines

CI/CD for machine learning isn't just CI/CD for software with a model file attached. ML systems have additional moving parts — data, models, features, and training pipelines — that all need to be versioned, tested, and deployed with the same rigor as application code. Here's how we build ML deployment pipelines that actually work.

Why Standard CI/CD Falls Short for ML

In traditional software, the code IS the system. Test the code, deploy the code, done. In ML systems:

• The model is a function of code AND data. The same training code with different data produces a different model. Both need to be versioned and tracked.
• Testing is probabilistic. You can't write a unit test that says "this model should return 42." You need statistical tests that verify performance across distributions.
• Deployment involves large artifacts. Model files can be gigabytes. You can't just push a new container image — you need artifact management for models, embeddings, and feature transformers.
• Rollback is complex. Rolling back a model might require rolling back the feature pipeline, the preprocessing code, and the serving configuration simultaneously.

The ML CI/CD Pipeline Architecture

We structure our ML pipelines into four stages:

Stage 1: Data Validation

Before any model training, validate that the input data meets expectations:

• Schema validation: Column types, required fields, value ranges. Use Great Expectations or a custom validation framework.
• Distribution checks: Compare incoming data distributions against training data baselines. Significant drift triggers an alert, not automatic retraining.
• Freshness checks: Verify that data sources have been updated within expected timeframes.
• Volume checks: Ensure you have enough data for meaningful training. A training run on 100 records because the pipeline silently failed is worse than no training at all.

Stage 2: Training and Experimentation

Automated training with proper experiment tracking:

• Every training run logs hyperparameters, data version, code version, and metrics to MLflow or Weights & Biases
• Training runs are reproducible — same code + same data = same model (within floating point tolerance)
• Compute is provisioned dynamically (e.g., spin up GPU instances for training, tear down after)
• Checkpointing protects against interrupted training runs

Stage 3: Model Testing

This is where most teams cut corners, and where most production failures originate. Our minimum test suite:

• Performance benchmarks: Accuracy, precision, recall, F1 (or relevant metrics) must meet minimum thresholds on a held-out test set.
• Regression tests: The new model must not perform worse than the current production model on a fixed benchmark set.
• Slice analysis: Performance must be acceptable across important data segments (demographic groups, product categories, geographic regions).
• Inference performance: Latency and throughput must meet SLA requirements under expected load.
• Model size and resource requirements: Verify the model fits within allocated GPU memory and compute budgets.
• Input/output contract tests: The model accepts the expected input schema and produces the expected output schema.

Stage 4: Deployment

Production deployment with safety nets:

• Shadow deployment: Run the new model alongside the production model, comparing outputs without affecting users. This catches issues that offline testing misses.
• Canary deployment: Route 5-10% of traffic to the new model and monitor key metrics. If metrics degrade, automatically roll back.
• Blue-green deployment: For simpler cases, swap traffic from old model to new model with instant rollback capability.
• Feature flags: Use feature flags to enable the new model for specific users, regions, or use cases before full rollout.

The Tools We Use

• Pipeline orchestration: Argo Workflows or Kubeflow Pipelines for defining and running multi-step ML pipelines
• Experiment tracking: MLflow for metrics, parameters, and model versioning
• Data versioning: DVC (Data Version Control) for tracking dataset versions alongside code
• Model registry: MLflow Model Registry for promoting models through staging to production
• Serving: Triton Inference Server or vLLM, deployed on Kubernetes with custom autoscaling
• Monitoring: Prometheus + Grafana for infrastructure metrics, custom dashboards for model performance metrics
• Alerting: PagerDuty integration for critical model failures, Slack for drift warnings and performance degradation

Common Mistakes

• Manual model deployment. If deploying a model requires SSH-ing into a server and copying files, you're one bad day away from a production incident. Automate everything.
• No rollback plan. Before deploying any model, verify you can roll back to the previous version in under 5 minutes.
• Testing only on average performance. A model with 95% average accuracy that drops to 60% for a specific customer segment is a production failure. Always analyze performance by segment.
• Ignoring data pipeline failures. The most common cause of model degradation isn't model rot — it's upstream data pipeline failures that go undetected.
• Treating model retraining as a batch job. Retraining should be triggered by monitoring (data drift, performance degradation) not by a calendar. Monthly retraining is arbitrary; retraining when the model needs it is engineering.