Python Packaging: Why Large Monorepos Build Their Own Package Manager

There’s a rite of passage in large Python monorepos. At some point — usually around the 100-package mark — someone on the platform team says “pip is too slow” and writes a shell script. The shell script grows. It learns about topological sorting. It gets a config file. Six months later, it has a name, a README, and a Slack channel for bug reports.

You’ve just accidentally built a package manager.

I’ve been working in a monorepo that went through this exact arc. The tool that came out the other side is called oaipkg, and the more I dug into how it works, the more I realized: this wasn’t a choice. It was gravity. The codebase got big enough that the gap between “what pip does” and “what we actually need” became a load-bearing problem. And the interesting part isn’t oaipkg itself — it’s why this keeps happening everywhere.

pip Has a Blind Spot, and It’s Shaped Like Your Repo

Every Python packaging tool — pip, uv, poetry, pdm — is built around one core assumption: your dependencies live on an index somewhere. PyPI, a private feed, doesn’t matter. There’s a registry, packages have version numbers, and the tool’s job is to solve a constraint satisfaction problem across those versions.

This assumption is like gravity in a video game. You don’t notice it until you try to walk on a ceiling.

In a large monorepo, maybe 40 of your dependencies are directories on disk. They’re not on any index. They don’t have version numbers — their “version” is whatever commit you’re on. And they depend on each other in a graph that changes every time someone merges a PR.

pip’s response to this is roughly: “Let me run setup.py in a subprocess for each of those 40 packages to figure out their dependencies.” So now you’re spawning 40 Python processes, each executing build machinery, just to answer the question “what depends on what.” For pure Python packages that need zero compilation. It’s like hiring a contractor to check if your light switch is on.

Two Graphs Walk Into a Repo

The fundamental problem is that a large monorepo has two dependency graphs, and they play by completely different rules.

The internal graph: Package A in libs/networking/ depends on Package B in libs/common/. There’s nothing to “resolve” — you always want whatever’s at HEAD. The question is topological order, not version compatibility. It’s a directory walk, not a SAT problem.

The external graph: Package A also depends on requests>=2.28. This is normal PyPI resolution. Versions, constraints, compatibility matrices. pip and uv handle this beautifully.

No standard tool understands both graphs at the same time. And the moment you try to force internal packages through the external-dep machinery, everything gets weird. pip tries to “resolve” version constraints that don’t exist. It downloads packages that are already sitting right there on disk. It builds wheels for things that don’t need building. It’s a category error, applied at scale.

What oaipkg Actually Does (and Doesn’t Do)

The architecture of oaipkg is almost disappointingly simple once you see it. The cleverness isn’t in any single piece — it’s in knowing which problems to solve yourself and which to delegate.

Static dependency discovery. Instead of executing setup.py to figure out what depends on what, oaipkg reads pyproject.toml files directly. Just TOML parsing. Internal deps live in a custom section:

[tool.oaipkg]
monorepo-dependencies = [
    "common",
    "networking",
    "auth-utils",
]

DFS walk across these declarations. Full internal dependency graph in milliseconds. No subprocesses.

The install method fallback chain. This is the part I find genuinely clever. Not every package is the same, so oaipkg picks the right install strategy per package:

1. Pure Python (90% of packages): Write a .pth file so Python can find it, generate a minimal .egg-info for importlib.metadata. Done. No pip, no build backend, no wheel. Instant.

2. Native extensions: Download a prebuilt wheel from Azure Blob Storage, keyed by a content hash of the source tree. CI built it; you just grab it.

3. Fallback: pip install -e. Slow, but always works.

Third-party delegation. Here’s the part that took me a while to get: oaipkg is not replacing pip or uv. It uses them. Once it resolves the internal graph, it collects all third-party deps, deduplicates them against a global constraints file, and hands the whole batch to pip/uv for resolution and installation.

oaipkg handles what pip can’t. Then it calls pip for what pip can.

The .pth File Trick Is the Whole Show

The editable install path deserves a closer look because it’s responsible for 90% of the speed gain, and it’s absurdly simple.

A .pth file is a one-liner that tells Python “also look in this directory for packages.” Drop a file called my-package.pth into site-packages with the path /repo/libs/my-package/src, and import my_package just works. No copying, no linking, no build step. Python’s import machinery handles the rest.

pip’s editable install eventually does something similar — but first it has to invoke a PEP 517 build backend, run the editable_wheel hook, possibly generate metadata, and then write the result. For one package that’s fine. For 40 packages it’s the difference between “blink and it’s done” and “go get coffee.”

oaipkg skips the ceremony. Read the TOML, write the .pth file, move on. The whole editable install for a pure Python package is basically open(path, 'w').write(src_dir). Everything else is bookkeeping.

Content-Addressable Wheels

For the packages that do need compilation — C extensions, Cython, anything that wraps native code — oaipkg has a trick stolen straight from Nix and Docker: content-addressable caching.

The problem: “has this native extension’s source changed since the last build?” Git commit hashes don’t help — a commit might touch files outside the package. Version bumps don’t help — monorepo packages often don’t even have real versions.

The solution: hash the actual file contents of the package’s source tree. Same files → same hash → same wheel. Change one .c file → different hash → rebuild.

CI computes the hash, builds the wheel, uploads {package}-{treehash}.whl to blob storage. At install time, oaipkg computes the same hash locally. Cache hit → download the wheel (parallel, 8 threads). Cache miss → build from source.

It’s the same insight behind git’s object model, Docker layer caching, and Nix store paths: address things by their content, not by arbitrary labels. Applied to Python wheels. Works beautifully.

The Scary Part: Auto-Install on Import

oaipkg has a meta path finder — a hook into Python’s import machinery — that can auto-install packages when you first import them. The codebase documentation calls this “the scariest thing oaipkg does,” which is exactly the kind of self-aware honesty I appreciate in infrastructure code.

You import a monorepo package. It’s not installed. The import hook catches the ImportError, runs the install (same fallback chain), and retries the import. From the developer’s perspective: the import just works, but it took 2 seconds the first time.

From the “I need to debug why my code is slow on first run” perspective: what the hell just happened.

Is it a good idea? Honestly, jury’s out. It’s incredibly convenient in dev. It’s terrifying in production. oaipkg wisely makes it opt-in.

Global Constraints: Democracy Is Overrated

In a monorepo with hundreds of packages, you can’t let each one pin its own numpy version. You’d end up with 47 different numpy versions across your dependency tree, and pip would either pick one and pray or give up entirely.

oaipkg uses a single constraints file that pins every third-party dependency for the entire repo. One numpy. One requests. One protobuf. For everyone.

The tradeoff is dictatorial: you can’t upgrade numpy in just your package. You upgrade it for everyone, or nobody. A committee decision on what the blessed version is.

In practice? This is the right call. The alternative — inconsistent versions across packages in the same repo — creates bugs that are genuinely impossible to diagnose. “Works on my package, breaks on yours, same numpy version string but different build flags” is a class of bug you encounter once and then immediately vote for the dictatorship.

The Bazel Surprise: It’s Barely Different

If you’re using Bazel in your monorepo — and at this scale, there’s a decent chance you are — you might expect the package manager to need deep Bazel integration. Something complicated. A rethinking of the install flow.

Nope. Same install flow. Same fallback chain. Same dependency resolution. The only differences: finding the monorepo root (environment variable instead of walking up to .git) and a .bazel suffix on wheel filenames to avoid collisions with non-Bazel wheels.

Bazel is just another build backend that produces wheels. oaipkg consumes wheels the same way regardless of who built them. The abstraction held.

Why Not Just Use uv?

This is the question. Everyone asks it. The answer is almost anticlimactic.

oaipkg already uses uv (or pip) as a backend. It’s not competing with them. It solves a problem that sits above them.

pip/uv answer: “Given package names and version constraints, find a compatible set, download, install.”

A monorepo package manager answers: “Given 300 directories of interdependent code, get 40 of them into a working virtualenv, pick the right install method for each, reuse cached builds, keep all third-party deps consistent — and then call pip to handle the PyPI stuff.”

The second question contains the first. The first doesn’t address the second at all.

It’s like asking “why do you need air traffic control? Planes already have pilots.” Yes. And the pilots are great. But they don’t solve the coordination problem.

The Pattern

Here’s what I find genuinely interesting about all of this: oaipkg isn’t unique. Google has one (parts of Blaze/Bazel). Meta has one (Buck’s Python support). Stripe, Uber, Airbnb — at a certain scale, they all end up building the same basic shape of tool. Static dep discovery, tiered install strategies, content-addressable caching, global constraints, delegation to standard resolvers for the PyPI graph.

It’s convergent evolution. The problem space is constrained enough that the solutions look similar, even when built independently. Which tells you something: this isn’t a tooling gap that better versions of pip will close. It’s a genuinely different problem. Standard package managers solve resolution. Monorepo package managers solve coordination. Same noun, different verb.

The Python packaging ecosystem is good — genuinely good, and getting better fast. uv has made single-project installs blazingly fast. But as your codebase grows past a certain threshold, the problem undergoes a phase transition. You go from “I have code that uses packages” to “I have packages that use packages that use packages, and they all live in the same repo, and they need to agree on everything at the same commit.”

At that point, you either build something like oaipkg, or you spend your days fighting tools that were designed for a world that doesn’t look like yours.

Either way, you end up with a package manager. The only question is whether you do it on purpose.