A model release includes more than model weights: code, a data snapshot, preprocessing, dependencies, and serving configuration all affect what reaches production. The pipeline below keeps that bundle traceable from validation to review, deployment, monitoring, and rollback. Its code and synthetic examples show the required handoffs; they are not evidence that a production deployment or training benchmark was run.
Machine-learning delivery is not ordinary CI with a larger build artifact. A code test can pass while a new data snapshot changes the feature distribution, labels arrive late, or a model loses calibration in production. A useful MLOps workflow connects code, data, evaluation, deployment, and monitoring so every promotion has evidence and a safe way back.
Key Takeaways
- CI tests code and data contracts; CD promotes a versioned artifact; continuous training proposes a new model.
- Every run should identify the code revision, data snapshot, parameters, metrics, dependencies, and artifact checksum.
- Offline quality gates and online monitoring answer different questions, so both belong in the release record.
- Automate candidate creation and checks first. Keep approval and rollback explicit for high-impact decisions.
CI, CD, and continuous training answer different questions
Continuous integration (CI) asks whether code and data contracts are valid. Continuous delivery (CD) packages and promotes a specific model artifact. Continuous training (CT) proposes a new candidate when data or outcomes justify a retrain. Combining the labels into one “automatic pipeline” hides where a failure occurred and who should approve it.
Follow one candidate through three different decisions
| Stage | Question | Output |
|---|---|---|
| CI: validate | Does the change satisfy code and data contracts? | Tests and a report; no production write. |
| CT: propose | Is the new candidate worth considering? | A versioned bundle and evaluation evidence. |
| CD: release | Is this exact bundle approved for this environment? | A controlled deployment and recovery record. |
The lifecycle: keep the evidence at each handoff
The stages below are a practical minimum. They can run in one GitHub Actions workflow or in a pipeline system, but the record should remain the same.
| Stage | Evidence to store | Gate question | Typical failure |
|---|---|---|---|
| Data validation | Schema, ranges, nulls, freshness | Can the training job consume this snapshot? | Missing column or shifted distribution |
| Training | Code SHA, parameters, environment | Can another worker reconstruct the run? | Untracked dependency or seed |
| Evaluation | Baseline, metrics, slices, policy checks | Does the candidate beat the declared floor? | Regression hidden by an average |
| Packaging | Model checksum and lockfile | Can deployment restore the exact bytes? | “Latest” tag moved |
| Canary | Traffic slice, latency, errors | Does the service behave online? | Resource or feature mismatch |
| Monitoring | Drift, calibration, outcomes | Has the operating context changed? | Silent decay or alert fatigue |
A release is only as reproducible as its least documented input. Store the data snapshot identifier, preprocessing code, dependency lockfile, random seed where meaningful, and evaluation configuration with the model artifact.
| Artifact | Example identifier | Why it matters |
|---|---|---|
| Code | git:4e2b9d1 | Use the full commit SHA in a real manifest; this is an abbreviated example. |
| Data | snapshot:2026-09-12T0100Z | Prevents a moving training target. |
| Environment | lock:sha256… | Record a real lockfile digest; the ellipsis shown here is not a valid checksum. |
| Model | registry:model-a@17 | Allows promotion and rollback by immutable version. |
| Evaluation | run:2026-09-13-042 | Keeps metrics, slices, and failed cases together. |
A small GitHub Actions gate
The earlier example did not transfer the candidate to the second job and attempted to run a script in a fresh runner without checking out the repository. The revised skeleton makes that boundary explicit. It validates a synthetic fixture on pull requests, and permits a reviewed release path only for a manual run on the main branch. No production deployment is configured in this sample.
name: ml-candidate-and-reviewed-release
on:
pull_request:
workflow_dispatch:
permissions:
contents: read
jobs:
evaluate:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
with: {persist-credentials: false}
- uses: actions/setup-python@v5
with: {python-version: "3.12"}
- run: python -m pip install --require-hashes -r requirements.lock
# Repository-owned scripts; use synthetic CI data, not production secrets.
- run: python tools/validate_data_contract.py --data fixtures/train.csv
- run: python tools/build_candidate.py --data fixtures/train.csv --output release/
- run: python tools/evaluate.py --candidate release/ --baseline metrics/baseline.json --report reports/evaluation.json
- name: Preserve diagnostic report, including gate failures
if: always()
uses: actions/upload-artifact@v4
with:
name: evaluation-report
path: reports/
if-no-files-found: warn
retention-days: 14
- name: Transfer only a successful candidate
uses: actions/upload-artifact@v4
with:
name: candidate-${{ github.sha }}
path: release/
if-no-files-found: error
retention-days: 14
reviewed_release:
if: github.event_name == 'workflow_dispatch' && github.ref == 'refs/heads/main'
needs: evaluate
runs-on: ubuntu-latest
environment: production
concurrency:
group: production-model-release
cancel-in-progress: false
steps:
- uses: actions/checkout@v4
with: {persist-credentials: false}
- uses: actions/setup-python@v5
with: {python-version: "3.12"}
- run: python -m pip install --require-hashes -r requirements.lock
- uses: actions/download-artifact@v5
with:
name: candidate-${{ github.sha }}
path: release/
- run: python tools/verify_manifest.py --input release/ --expected-commit "${{ github.sha }}"
- name: Stop until the deployment adapter is configured
run: |
echo "No deployment configured. Add reviewed auth, canary, and rollback steps."
exit 1
Prerequisites: Supply the named repository-owned scripts, fixture, baseline, and a fully pinned hash-checked requirements file. The build must write a manifest containing the source commit, data identifier, environment and model checksums. The evaluator must exit nonzero on a failed gate; the verifier must reject a mismatch. These script names describe contracts, not bundled implementations. The YAML was parsed and its event/job conditions inspected; no cloud deployment or paid training run was executed.
The final step intentionally fails closed. Replace it only after implementing and reviewing the provider-specific authentication, target environment, canary, outcome check, and rollback adapter. Prefer narrowly scoped short-lived credentials where supported; an OIDC login still requires an explicitly configured trust policy. The pull-request job should have no production credentials, privileged self-hosted runner, or side-effecting registry access.
Configure required reviewers and allowed deployment branches in the actual GitHub environment; writing environment: production does not create those protections by itself. Availability of protection options depends on the repository and plan. Pin third-party actions to reviewed full commit SHAs for production. The readable major-version tags here are teaching references, not a claim to be the latest releases. GitHub deployment controls.
Upload/download artifacts provide an explicit handoff between runners; a dependency cache is not a release bundle. Retain the diagnostic report even when evaluation fails, and publish a candidate artifact only after a successful gate. Keep durable approved releases outside short-lived workflow retention. GitHub artifact handoff.
Quality gates need a declared baseline
Do not choose thresholds after seeing the candidate result. Define a baseline model, a minimum quality floor, and any slice-specific policy checks before training. An average metric can improve while a small but important segment regresses, so record both the aggregate and the slices that matter to the product.
Every metric keeps its own scale
| Gate | Baseline → candidate | Illustrative acceptance rule |
|---|---|---|
| Macro F1 ↑ | 0.82 → 0.85 | Candidate ≥ 0.83; also inspect named slices. |
| Calibration error ↓ | 0.12 → 0.09 | ECE ≤ 0.10 with fixed bins and the same evaluation set. |
| Service p95 latency ↓ | 220 → 205 ms | ≤ 250 ms under the same load test. |
| Critical slice recall ↑ | 0.88 → 0.90 | ≥ 0.89 for the predeclared slice. |
F1 and recall: higher is better. ECE and latency: lower is better. Each axis is labeled independently; F1 and recall use a shortened range. Fix ECE bins, evaluation data, load, and slice definitions before testing.
The figure is synthetic, not a measured result. It keeps F1, expected calibration error (ECE), latency, and slice recall separate instead of putting incompatible units on normalized bars. Fix the evaluation split, ECE binning, load-test conditions, and slice definitions before comparison. Include sample counts and uncertainty when drawing conclusions from real data; a tiny apparent improvement is not necessarily meaningful.
Monitoring turns a deployed model into an observable service
Offline evaluation cannot predict every change in traffic. Track input drift, prediction distribution, error rate, latency, and a delayed business outcome such as approval quality or fraud loss. Define the owner, window, and action for each alert. A threshold without a response is decoration.
An alert needs an owner and an action
| Synthetic signal | Window / rule | Response |
|---|---|---|
| Request errors: 1.8% | 15 minutes; investigate above 2% with a minimum traffic count. | Serving owner inspects errors and recent changes. |
| p95 latency: 205 ms | 15 minutes; service objective 250 ms. | Serving owner checks load and resource saturation. |
| Label lag: 6 days | Daily; review when older than 7 days. | Data owner investigates missing outcomes before retraining. |
| Feature PSI: 0.07 | Daily; example review threshold 0.10. | Data scientist checks fixed bins and affected segments. |
Near-limit signals should create a candidate review or a controlled investigation. Automatically replacing production on every alert creates an unbounded change generator and makes it difficult to tell whether the new model caused the incident.
Retraining and rollback
Retraining can be automated safely when it produces a candidate, runs the same evaluation suite, and records the data and code used. Promotion still needs a policy decision for models that affect money, access, safety, or regulated decisions. A rollback is real only when the previous model, feature definitions, dependencies, and configuration are still available.
A rollback restores a release bundle
Model bytes + checksum
Feature schema v6
Preprocessor v6
Runtime lock B
Serving configuration B
Model bytes + checksum
Feature schema v5
Preprocessor v5
Runtime lock A
Serving configuration A
Check schema compatibility before switching traffic. A model-only rollback can fail if live features still use the candidate’s schema.
A rollback drill before the incident
Use a staging service and a synthetic request set. First deploy the known-good bundle and record its outputs. Deploy the candidate, then introduce a controlled failure such as a deliberately incompatible feature schema. Check that the release is stopped and that the operator can restore the prior bundle without guessing which dependencies were used.
After restoration, verify the served model version, feature/preprocessing compatibility, health checks, and representative output. Do not stop at “the rollback command exited successfully.” Record recovery time and any manual repair. A checksum detects a byte mismatch; it does not identify a trustworthy publisher by itself. If an attacker can replace both the artifact and its checksum record, that check alone is insufficient.
A drift alert is a reason to investigate, not a diagnosis of model decay. A changed input mix, a broken data feed, and genuinely changed relationships can look similar on one dashboard. Delayed labels can make apparent accuracy stale. Keep the alert, data investigation, candidate experiment, and production decision as separate records.
What this workflow does not guarantee
- It does not guarantee fairness: slice metrics and review criteria must be defined for the domain.
- It does not make data stationary: drift detection identifies a change; it does not explain its cause.
- It does not remove human ownership: a person or named team still owns promotion, incident response, and deletion decisions.
- It does not make “latest” reproducible: use immutable model, data, and dependency identifiers.
Editorial Verdict
A good ML CI/CD system makes a model release boring: the data snapshot is known, checks are repeatable, the artifact is immutable, monitoring has an owner, and rollback is one documented action. Build that chain before adding more automation.
Frequently Asked Questions
Is CI/CD enough for machine learning?
No. You also need data contracts, reproducible inputs, slice-aware evaluation, outcome monitoring, and a retraining decision. Passing software tests does not prove that the training data is appropriate or that the model remains useful.
When should retraining be automatic?
Automate candidate creation and evaluation when the inputs and costs are controlled. Promotion is a separate decision with a named owner and the required approvals. Do not let every alert silently replace production.
Why does the second job download an artifact?
Each hosted job starts on a separate runner. A file produced by the first job is not automatically present in the next one. Transfer the exact candidate explicitly, then verify its manifest before considering deployment.
Does the example deploy a working model?
No. It is an integration skeleton requiring your scripts and environment configuration. Its last step deliberately stops until a reviewed deployment adapter exists. That is safer and more honest than a green job that appears to have deployed nothing.
Does a random seed make training perfectly reproducible?
Not necessarily. Libraries, hardware, parallel execution, data order, and nondeterministic operations can affect results. Preserve the actual model bytes and environment, and state the level of reproducibility you tested.
What is the smallest useful rollback plan?
Retain a known-good model together with compatible preprocessing, feature definitions, dependencies, and serving configuration. Rehearse the switch and verify outputs. A moving “latest” tag or an expired workflow artifact is not a recovery plan.
Official Sources
- GitHub Actions deployment guide — workflow triggers, environments, and deployment controls.
- MLflow Tracking — runs, parameters, metrics, and artifacts.
- TensorFlow Extended guide — data validation and model-analysis pipeline concepts.
- DVC documentation — versioning data and reproducible experiments.
Tools and APIs change. The workflow and deployment references were rechecked on September 19, 2026. Confirm current syntax, action compatibility, retention, and permissions before adapting the sample.
Editorial record
Editorial update — September 19, 2026: Fixed the cross-job artifact handoff, separated pull-request validation from production authority, added a fail-closed deployment placeholder, and replaced mixed-unit charts with explicit gates and a rollback manifest. Original publication date retained. Figures use synthetic teaching data.