A uv sync that takes 10 seconds on your laptop was taking 40 minutes on our GPU cluster. Here’s why.
Local Disk vs Network Filesystem
Your laptop has an SSD. File operations — creating files, reading metadata, writing .pyc — take microseconds.
GPU clusters don’t work like that. Nodes are stateless. If a node dies, another one picks up. So you need a shared filesystem — something every node can see. Ours is called VAST. Think of it like a giant NFS mount at /mnt/vast.
Training data, checkpoints, container images — they all live on VAST. That’s the point. Multi-node jobs need to read the same data, and you can’t put 50TB of training data on every node’s local SSD.
The Problem
uv sync --package rocket --frozen installs 64+ Python packages. That means thousands of small file operations — creating virtualenv directories, writing package metadata, compiling .pyc files, building C extensions.
On local SSD: seconds. Each file op is microseconds.
On a network filesystem: each file operation pays a network round-trip. Multiply microseconds by a network hop, then multiply by thousands of files. 40 minutes.
The fix is obvious once you see it — run uv sync on local disk (/tmp), not the shared filesystem. The venv doesn’t need to be visible to other nodes. Only training data does.
Same Energy: Docker Registry Cache
Related gotcha with Docker builds in CI.
Docker layer caching is great — if a layer hasn’t changed, skip it. But on ephemeral CI runners (no persistent local disk), “cached” just means “I won’t re-execute the command.” The layer blobs still need to be downloaded from the registry and extracted to assemble the image.
We had a COPY --from=some-image /hf_home/ /hf_home/ step copying ~1.5GB of HuggingFace model weights. BuildKit says “CACHED” — but then spends 7 minutes downloading and extracting the blobs anyway. Because there’s no local layer store. Every run starts from scratch.
The word “cached” is doing a lot of heavy lifting. It means “skip re-execution,” not “skip the work.”
The Pattern
Both problems are the same thing: assuming file I/O is cheap when it isn’t.
- Network filesystem: each small file op has latency you don’t notice on local disk
- Registry cache on ephemeral runners: “cached” still means downloading gigabytes
When you see something taking way longer than expected, ask: where are the files actually living, and what’s the cost per operation?