Caching Layers for a Python Monorepo: It’s Not One Problem

Our CI caching strategy used to be a single line: actions/cache. Then the monorepo grew past the point where 10GB covers anything, and now we’re designing a multi-layer cache architecture like it’s an L1/L2/L3 situation. Because it literally is:

• L1 (uv cache / repo cache on PVC) – fast, per-core (per-runner), small. Always local, zero overhead.
• L2 (proxy registry) – slightly slower, shared across cores (cluster-wide), larger. One miss warms the cache for everyone.
• L3 (GHA actions/cache) – slowest, shared across everything, size-limited, and evicts aggressively. The one everybody starts with and the one that breaks first.

This post captures where we are in that thinking. Not a solved problem – a snapshot of the design space our DevEx squad is working through for the yolo monorepo. If you’re running CI for a Python monorepo and wondering why actions/cache keeps evicting the one thing you actually needed, this might resonate.

For the Bazel-specific version of this problem, see Bazel CI Cache: Why GitHub Actions Cache Won’t Save You. For the runner infrastructure that makes most of this possible, see The CI Runner Landscape.

The setup

Our repo (yolo) is a Python monorepo. uv workspaces, one big uv.lock, Bazel migration in progress, Docker images built for everything, CI on GitHub Actions with self-hosted ARC runners on Kubernetes. ACR for images.

The caching problem breaks down into three things that are slow when cold:

1. Package resolution – uv downloading packages from PyPI
2. Repository checkout – git cloning a monorepo with long history
3. Everything else – Docker layers, Bazel artifacts, etc. (covered in other posts)

Each of these has a different optimal caching strategy. Trying to solve them all with one mechanism is how you end up with a 10GB actions/cache that’s 40% stale npm artifacts and 60% regret.

Layer 1: uv cache on a persistent volume

uv keeps a local cache of downloaded packages at ~/.cache/uv (or wherever UV_CACHE_DIR points). When the cache is warm, uv sync skips downloading anything it already has. Depending on your dependency tree, this is the difference between a 5-minute install and a 15-second one.

On ephemeral GitHub-hosted runners, this cache disappears after every job. You can use actions/cache to persist it, but you’re fighting the 10GB limit and spending time on compress/upload/download cycles.

On self-hosted runners, the answer is simpler: mount a PersistentVolumeClaim to the cache directory.

# ARC runner pod spec (simplified)
containers:
  - name: runner
    volumeMounts:
      - name: uv-cache
        mountPath: /home/runner/.cache/uv
volumes:
  - name: uv-cache
    persistentVolumeClaim:
      claimName: uv-cache-pvc

No upload. No download. No compression. The cache is just there when the pod starts. The volume persists across pod restarts, and uv handles its own cache invalidation – new versions of packages get added, old ones sit around until you prune.

The tradeoff: this is per-runner. If you have 20 runner pods, you have 20 copies of the uv cache. Each one warms up independently. That’s fine for steady-state (they all converge on the same packages eventually), but it means the first few runs on a fresh pod are still cold.

This is the same “cloud disk” strategy that works for Bazel’s disk cache, just applied to a different tool. The principle is identical: if you own the runner, skip the cache-as-a-service layer entirely and use a disk.

Layer 2: proxy registry inside the cluster

The per-runner uv cache solves the “don’t re-download packages I already have” problem for a single runner. But what about the first run on a new runner? Or when a new dependency shows up and 20 runners all need it at the same time?

That’s where a proxy registry comes in. Run a PyPI proxy (devpi, Artifactory, Azure Artifacts, whatever) inside the Kubernetes cluster. Point uv at it:

# pyproject.toml or uv.toml
[tool.uv]
index-url = "https://pypi-proxy.internal.cluster/simple"

When uv requests a package, the proxy checks its own cache first. Cache hit: serve it from local storage, no external network call. Cache miss: fetch from upstream PyPI, cache it, serve it.

This is a cluster-wide cache. Every runner benefits from it. The first runner to need torch==2.5.0 pulls it from PyPI; every subsequent runner gets it from the proxy at local network speed. For a 2GB package like torch with CUDA extensions, that’s a meaningful difference.

Other benefits that aren’t obvious until you need them:

• PyPI outage protection. PyPI goes down more often than you’d think. A local proxy with cached packages means your CI keeps running. Not forever – but long enough for PyPI to come back.
• Network cost reduction. If your cluster is in Azure and PyPI is somewhere else, you’re paying egress for every download. A proxy turns N downloads into 1.
• Audit trail. You can see exactly what packages your CI is pulling, and when. Useful when security asks “are we using any packages from that compromised maintainer?”

We’ve written about the complexity of PyPI registries in ADO before. The proxy registry is a simpler version of the same idea – less about access control, more about not hitting the network when you don’t need to.

Layer 3: repository cache on disk

This one is easy to overlook because actions/checkout handles it transparently. But in a monorepo with years of history, git clone is not free.

On ephemeral runners, every job clones from scratch. You can use fetch-depth: 1 for a shallow clone, which helps, but some tools (Bazel with git_override, bazel diff, blame-based workflows) need the full history.

On self-hosted runners with persistent disk, you can keep a local clone and just git fetch on each run:

- uses: actions/checkout@v4
  with:
    path: /opt/repo-cache/yolo
    clean: false  # don't wipe the directory

Incremental fetch is dramatically faster than full clone. For our repo, that’s the difference between 90 seconds (clone) and 5 seconds (fetch). Over thousands of CI runs per day, it adds up.

Same tradeoff as the uv cache: per-runner, not shared. But unlike package caches, git is smart about this – a git fetch only transfers new objects, and the git protocol is efficient about deltification.

Layer 4: GitHub Actions cache (the one with limits)

And then there’s actions/cache. The one everybody starts with, and the one that works fine until it doesn’t.

The constraints:

For a monorepo with 50 active branches and 10 workflows, 10GB is nothing. You’re constantly fighting eviction. The main branch caches get pushed out by feature branch caches, which themselves get pushed out by other feature branches. It’s musical chairs with 50 players and 3 chairs.

The LRU eviction is particularly painful for infrequent-but-expensive builds. If you have a weekly integration test that takes 30 minutes with a cold cache and 5 minutes with a warm one, but the cache gets evicted by day 7 because nothing accessed it… you’re rebuilding from scratch every single week.

actions/cache is fine for small things – a pip cache for a single-package repo, node_modules for a frontend project. But for a monorepo running CI at scale, it’s a stopgap, not a solution.

The scoping architecture

The thing that made this click for me: these aren’t just different caching mechanisms. They’re nested scopes. Each layer wraps the one inside it, and the scope determines who benefits:

┌─────────────────────────────────────────┐
│  Proxy Registry (cluster-wide)          │  ← Every runner pod benefits
│  ┌───────────────────────────────────┐  │
│  │  Runner Pod (per-runner)          │  │
│  │  ┌─────────────┐ ┌────────────┐  │  │
│  │  │ uv cache    │ │ repo cache │  │  │  ← PVC mounts, per-pod
│  │  │ (PVC mount) │ │ (PVC mount)│  │  │
│  │  └─────────────┘ └────────────┘  │  │
│  └───────────────────────────────────┘  │
└─────────────────────────────────────────┘

Cluster-wide → per-runner → per-job. The proxy registry is the outermost layer – anything cached there benefits every runner in the cluster. Inside that, each runner pod has its own PVC-backed caches that persist across jobs on that specific pod. And at the innermost level, each individual job run has whatever’s already warm on its assigned runner.

This is why actions/cache feels so awkward at scale – it’s trying to be a cluster-wide cache (shared across workflows) but with per-repo limits and aggressive eviction. It doesn’t fit neatly into any of these scopes. The three layers above are architecturally clean because each one has a clear scope boundary.

The layered picture

Here’s where we landed in the meeting: caching isn’t a single mechanism, it’s a stack.

The first three require self-hosted runners. That’s the fundamental unlock. If you’re on GitHub-hosted runners, you’re stuck with layer 4 and whatever cleverness you can squeeze into 10GB.

With self-hosted ARC runners on Kubernetes, the per-runner caches (layers 1 and 3) are just PVC mounts. The proxy registry (layer 2) is a service running in the cluster. None of this requires actions/cache at all.

The layers also have different warming characteristics:

• Proxy registry warms globally – one runner’s miss becomes every runner’s hit
• PVC caches warm per-runner – each pod builds up its own local state
• GHA cache warms per-key – and evicts aggressively under pressure

In practice, you want the proxy registry for the “global warming” effect and the PVC caches for the “instant access, zero overhead” effect. GHA cache becomes a fallback for things that don’t fit neatly into the other layers.

The self-hosted assumption

Self-hosted runners don’t just give you faster builds. They give you access to the entire caching stack.

All of this assumes you own your runners. That’s not a small assumption – it means running ARC, managing Kubernetes, monitoring disk usage, handling runner scale-up/scale-down. There’s a whole infrastructure progression that gets you there.

But once you’re there, caching becomes an infrastructure problem instead of a GitHub configuration problem. And infrastructure problems, annoyingly, are the kind you can actually solve.

If you’re not there yet – if you’re still on ubuntu-latest and your actions/cache is holding on by a thread – the proxy registry is the one thing you can deploy independently. It doesn’t require self-hosted runners. Point UV_INDEX_URL at your proxy, and every runner (hosted or self-hosted) benefits from the cluster-wide package cache. That’s the highest-leverage single change.

What we haven’t decided yet

This is a living problem. Some things we’re still figuring out:

• PVC sizing and cleanup. How big do the uv and repo cache PVCs need to be? When do we prune? uv has uv cache clean, but when do you run it? Separate CronJob? After N builds?
• Cache warming for new runners. When ARC scales up and spawns a fresh pod, that pod’s PVC caches are cold. Do we pre-warm from a snapshot? Just let it warm naturally? How many cold-start runs are acceptable?
• Proxy registry choice. devpi is simple but doesn’t do much beyond caching. Artifactory does everything but costs money and has its own operational burden. Azure Artifacts integrates with our stack but has its own quirks.
• Interaction with Docker layer caching. uv’s cache and Docker’s build cache are related but separate problems. A warm uv cache inside a Docker build (via --mount=type=cache) is different from a warm uv cache on the runner. We’ve written about the uv.lock cache bust problem and splitting uv.lock for faster Docker builds – those are downstream of this caching discussion.

The theme is: every layer helps, no single layer is enough, and the real optimization is picking the right layer for each workload instead of cramming everything into actions/cache and hoping for the best.