ML
Ask any ML engineer what version of a model is currently serving a production endpoint and you will get one of three answers: a confident answer backed by a registry, a hesitant answer backed by a file timestamp, or a blank stare. The third answer is more common than it should be. Effective model versioning is foundational to safe operations, reproducible science, and fast incident response.
Why Model Versioning Is Different From Code Versioning
Code versioning is solved: git provides a complete, immutable history of every change to every file. Model versioning is harder for several reasons. Model artifacts are large binary files that do not diff well. A model version is the product of code, data, and hyperparameters, so you need to version all three to reproduce it. And unlike code, where the artifact is deterministic given the source, training introduces randomness that means the same source may not produce the same artifact twice.
This means you need a versioning strategy that tracks not just the artifact itself but its provenance: what code was used to produce it, what dataset it was trained on, what hyperparameters were set, and what evaluation results it achieved. The artifact alone is not enough.
Semantic Versioning for ML Models
Semantic versioning as used in software (MAJOR.MINOR.PATCH) maps reasonably well to ML models with some adaptation. A useful convention is: MAJOR version for architecture changes that break the input/output interface; MINOR version for retraining on new data or significant hyperparameter changes; PATCH version for minor config updates, calibration fixes, or threshold adjustments that do not affect the model weights.
The key discipline is consistency. If a MINOR version bump always means "retrained on data through date X," then everyone on the team understands what a MINOR bump signifies without reading the release notes. Semantic versioning is most valuable as a communication tool, not just an identifier.
The Model Registry as Single Source of Truth
A model registry is the central database of all model versions, their metadata, and their deployment status. Every trained model artifact should be registered before it is eligible for deployment. Every deployment should reference a registry entry, not a file path. If a model exists only on a shared filesystem or in an S3 bucket without a registry entry, it is invisible to your governance, rollback, and audit tooling.
The minimum metadata a registry entry should capture is: model name and version, artifact location (URI, hash), training code commit, training dataset version, evaluation metrics, training date, author, and current lifecycle stage (candidate, staging, production, archived). With this metadata, you can answer any question about what is deployed, when it was trained, who trained it, and how it performed at training time.
MLPipeX's built-in model registry enforces metadata completeness at registration time. You cannot register a model without providing the training commit hash, dataset reference, and evaluation results. This discipline pays dividends every time you need to investigate a production issue.
Lifecycle Stages and Promotion
A model version should move through defined lifecycle stages: typically candidate (just registered), staging (passing automated tests), production (serving live traffic), and archived (retired). Promoting a model from one stage to the next should require explicit action, not happen automatically. This gives the team a clear view of what is deployed and what is waiting, and prevents accidental promotions.
The number of stages depends on your organization's risk tolerance. A startup shipping a recommendation model might use two stages: candidate and production. A financial services team shipping a credit scoring model might use five stages with review gates at each transition. The stages should match your actual review process, not an aspirational one that no one follows.
Managing Multiple Active Versions
Production systems sometimes need to run multiple versions of a model simultaneously. A/B testing requires two versions receiving traffic. Shadow deployments run a new version alongside the current one. Regional rollouts may have different versions in different geographies. Your versioning and deployment system needs to support this explicitly, not as a workaround.
In MLPipeX, an endpoint can be configured as a multi-variant endpoint with traffic split rules. Each variant references a specific model registry entry. Metrics are collected per variant so you can compare performance directly. Promoting one variant to 100% traffic and retiring the others is a single configuration change.
Artifact Immutability
A versioned model artifact must be immutable. Version 1.2.3 should always refer to the same weights, forever. If you can overwrite version 1.2.3 with different weights, your version numbers are meaningless as identifiers and your rollback capability is broken. Enforce immutability at the storage layer: use object storage with versioning and delete protection enabled, and never allow artifacts to be updated in place.
This seems obvious but is frequently violated in practice. The failure mode is subtle: an engineer "fixes" a bug in a registered model by overwriting the artifact without bumping the version. The registry says v1.2.3 is in production. The endpoint is now serving different weights than what v1.2.3 in the registry describes. Your audit trail is corrupted and you do not know it.
Archiving and Retention
Model artifacts are large. Keeping every version indefinitely is expensive. A retention policy that archives versions out of production for more than 6 months to cold storage, and deletes versions out of production for more than 2 years, is reasonable for most teams. Always keep at least the two previous production versions immediately available for fast rollback before archiving them.
The registry entry should be retained even when the artifact is deleted. The metadata about what a version was trained on and what it achieved is valuable for audit and reproducibility even if the artifact itself is no longer available.
Conclusion
Model versioning done well is invisible: deployments are auditable, rollbacks are instant, and incidents are diagnosable. Model versioning done poorly is a constant source of friction and risk. The investment is not large. A registry, a semantic versioning convention, and immutable artifact storage are sufficient to build on. Start there and add lifecycle stages and multi-version support as your team's complexity grows.