Reproducible GPU environments: why ML setups rot, and how to stop it

A diffusers pipeline that ran fine on Monday threw an ImportError on Thursday. Nobody touched the code. Nobody changed the pins. The notebook was identical to the run that worked three days earlier. It just stopped importing.

This happens to someone every week, and it is almost never their fault. It is a packaging problem, and it is worth understanding, because the fix is the same one the rest of software settled on years ago and GPU rentals mostly skipped.

What actually broke

Here is the mechanism. The user had pinned diffusers to an exact version. What they had not pinned, because almost nobody does, was huggingface_hub, a library diffusers depends on and imports. Somewhere between Monday and Thursday, huggingface_hub published a new release that dropped a helper function diffusers was still calling. The next time the container built its environment, pip resolved huggingface_hub to that newest release, because nothing told it not to, and the import chain snapped.

The code was frozen. The environment was the variable. That is the whole bug.

Pinning torch==2.5 doesn't pin anything

When you write torch==2.5.1 in a requirements file, you have pinned exactly one package. Installing it pulls in numpy, sympy, networkx, filelock, fsspec, a stack of nvidia-cu* CUDA wheels, triton, and each of their dependencies in turn. Add diffusers, transformers and accelerate and a realistic fine-tuning setup resolves to well over a hundred packages. A saved environment we captured last week locked 146 of them.

You named one. pip chose the other 145, live, at install time, using whatever was the newest compatible release on PyPI that day.

Those unnamed packages are your transitive dependencies: the dependencies of your dependencies. Each has its own maintainers and its own idea of what counts as a breaking change. torch never moved. One package three levels below it did, and that was enough to take the whole environment down.

"It worked last week" is the same bug as "it works on my machine"

The two complaints are one problem separated by time instead of by machine. In both, the source is fixed and the environment underneath it is quietly different. On your laptop it is different because your colleague resolved their packages on a different day. Weeks later it is different because you would resolve them on a different day.

Version numbers do not save you here. Semantic versioning is a promise maintainers make, not a guarantee the installer enforces, and a lot of the ML ecosystem lives below 1.0, where the convention explicitly promises nothing about stability. A 0.24 to 0.25 bump is free to delete a function. And even a careful maintainer's "minor" release can break code that leaned on behaviour they never documented.

The tempting fix is to pin more packages by hand. That does not scale. The transitive set is too large to track, it reshuffles the moment any direct pin changes, and you would end up maintaining a two-hundred-line file describing an install you never actually ran.

What reproducibility actually requires

Three things, and you need all three.

First, a real lockfile with hashes. Not the list of packages you asked for, but the full list of what actually gets installed, with every package pinned to an exact version and to the SHA-256 of the exact file. Now the installer has no room to improvise. If a downloaded wheel does not match its hash, the install fails loudly instead of quietly running something you never tested. This is what npm, cargo, poetry and uv all produce, and it is the part most GPU platforms skip.

Second, a durable copy of the artifacts. A lockfile that points at PyPI stays reproducible only for as long as PyPI keeps those exact files. Maintainers yank releases. Projects get deleted. A hash you cannot download is a receipt for a package you can no longer install, so real reproducibility means keeping the wheels themselves somewhere you control.

Third, continuous verification. A lockfile proves an install is byte-identical to last time. It does not prove the install still works: that the CUDA wheels still load against the driver on the box, that nothing in the catalog rotted for a fresh environment resolving today. The only way to know is to install it and import it on a schedule, and find out before a user does.

How Clodei does each

The versions never drift. Every image is built from a lockfile with hashes, and every add-on you pick at launch is resolved against that same lock and verified by SHA-256 before it installs. The stack you tested is the stack that runs. No surprise upgrade slips in between launches.

It still works a year later. We keep a private mirror of the exact wheels every environment uses, addressed by their content hash, on our own storage rather than PyPI. Save an environment today and it rebuilds identically next year, even if those exact versions have since been yanked or pulled from the public index.

It is checked every week. CI installs and imports every package in the catalog on the real GPU images, weekly, plus a monthly pass that proposes version bumps as reviewable pull requests. When something breaks upstream, it breaks in our pipeline first, not in your notebook.

Saved environments capture what actually happened. When you save an environment, we record the versions that genuinely installed, the resolved set rather than the short list you requested, and relaunch from exactly that. We byte-compare the delivered lock against the saved one before it ships, so a relaunch matches down to the file.

The honest part

None of this is clever. Lockfiles with hashes, a mirror of your own artifacts, and a CI job that reinstalls everything are ordinary supply-chain hygiene. Application developers have had it for years. What is genuinely uncommon is finding it applied to a GPU you rent by the minute, where the normal experience is pip install against a moving target and a shrug when it breaks.

An environment that quietly rots is not the cost of doing ML work. It is a packaging decision someone made for you, and it can be made differently. The environment you save should be the environment you get back, this week and a year from now.