Python Packaging: pyproject.toml 101

A teammate opened a PR that added three packages to a dependency group in our root pyproject.toml. I looked at it and realized I didn’t actually understand what that meant. Like — I knew what pyproject.toml was in the same way I know what a carburetor is. It exists. It does something important. I should probably not touch it.

So I went and figured it out.

The file that replaced everything

Python used to have setup.py, setup.cfg, requirements.txt, MANIFEST.in, and sometimes a Makefile thrown in for good measure. Five files doing what should be one file’s job. pyproject.toml is that one file.

It’s defined by a series of PEPs (517, 518, 621, 735 — don’t memorize these), and it’s now the standard way to configure a Python project. Build system, dependencies, tool configuration, metadata — all in one place.

If you come from JavaScript, it’s package.json. If you come from Rust, it’s Cargo.toml. If you come from the old Python world, it’s setup.py + requirements.txt + setup.cfg fused together and given a proper format.

The anatomy

A minimal pyproject.toml has three sections that matter:

[project]
name = "my-package"
version = "1.0.0"
requires-python = ">=3.10"
dependencies = [
    "requests>=2.28",
    "pydantic>=2.0",
]

[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"

[tool.ruff]
line-length = 100

[project] — Your package identity and its runtime dependencies. Name, version, what Python versions it supports, what it needs to run. This is the stuff that pip/uv reads when someone installs your package.

[build-system] — How to build the package into a wheel. hatchling, setuptools, flit — pick one. This tells pip which tool to invoke when building, not how to configure that tool.

[tool.*] — Configuration for development tools. Ruff, pyright, pytest, black, isort — instead of having a ruff.toml, a .pyrightconfig.json, and a pytest.ini, you can put them all under [tool.ruff], [tool.pyright], [tool.pytest.ini_options]. One file to rule them all.

That’s the basics. Now let’s talk about the part that confused me.

Three kinds of dependencies (and they’re all different)

This is where it gets interesting. pyproject.toml has three different ways to declare dependencies, and they serve completely different purposes.

1. dependencies — what your package needs to run

[project]
dependencies = [
    "requests>=2.28",
    "pydantic>=2.0",
]

These are your runtime deps. If someone pip installs your package, these get installed too. Non-negotiable. This is the equivalent of "dependencies" in package.json.

2. optional-dependencies — extra features, opt-in

[project.optional-dependencies]
gpu = ["torch>=2.0", "torchvision"]
viz = ["matplotlib", "plotly"]

These are extras that only get installed if someone asks for them: pip install my-package[gpu]. The square bracket syntax. You’ve probably seen it — pip install requests[security] or pip install fastapi[all].

Good for when you have heavy dependencies (like torch at 2GB) that not every user needs. The JS equivalent would be… there isn’t a clean one. Maybe peerDependencies, conceptually.

3. dependency-groups — development profiles (PEP 735)

[dependency-groups]
dev = [
    "pytest>=8.0",
    "ruff==0.9.9",
    "pyright==1.1.408",
]
notebook = [
    "jupyterlab>=4.4.3",
    "ipykernel>=6.29.5",
]
experimental = []

This one’s newer — PEP 735, adopted in late 2024. Dependency groups are not part of your package. They don’t get published to PyPI. They don’t affect anyone who installs your package. They’re purely for development.

Think of them as install profiles. dev is your linters and test runners. notebook is your Jupyter setup. experimental is… whatever you put in it.

The mental model: dependencies = what your users need. optional-dependencies = what your users might need. dependency-groups = what your developers need. They live in different sections because they serve different audiences.

The JS analogy: dependency-groups is like devDependencies in package.json — but more granular. Instead of one bag of dev deps, you can have as many named groups as you want. And then this line controls what you get by default:

[tool.uv]
default-groups = ["dev", "condor2"]

Run uv sync and you get dev and condor2. Want notebooks? uv sync --group notebook. Want the experimental stuff? uv sync --group experimental. Everyone gets exactly what they need, nothing more.

The monorepo twist

Everything above applies to a single package. A monorepo is a different animal.

Our repo — yolo, a Python monorepo for training multimodal LLMs — has 80+ packages. mai_config for configuration, mai_layers for model layers, mai_trainer for training loops, rocket for inference, plus a constellation of libraries, apps, and data processing tools.

Each package has its own pyproject.toml:

# mai_config/pyproject.toml
[project]
name = "mai-config"
version = "0.2.7"
dependencies = [
    "omegaconf>=2.3.0",
    "hydra-core>=1.3.2",
    "torch>=2.7.0",
    "pydantic>=2.0.0",
]

But there’s also a root pyproject.toml that ties everything together. And this one is special.

The coordinator pattern

# Root pyproject.toml
[project]
name = "yolo"
version = "0.2.0"

[tool.uv]
package = false

[tool.uv.workspace]
members = [
    "mai_config",
    "mai_layers",
    "mai_trainer",
    "rocket",
    "experimental",
    # ... 80+ more
]

package = false is the key line. This root pyproject.toml is not a Python package. You can’t pip install it. It doesn’t produce a wheel. It’s purely a coordination layer — it defines the workspace and provides shared configuration.

If you know npm/yarn/pnpm workspaces, this is the exact same pattern. The root package.json in a JS monorepo isn’t a publishable package either — it just declares "workspaces": ["packages/*"] and holds shared devDependencies.

One lockfile to rule them all

When you run uv lock in a workspace, uv resolves every package’s dependencies together and produces a single uv.lock. Ours is ~18,000 lines. One lockfile ensures that if mai_config needs pydantic>=2.0 and mai_trainer needs pydantic>=2.5, they both get the same resolved version. No conflicts at runtime, no “works in package A but breaks in package B.”

Cross-package dependencies

Packages within the workspace can depend on each other:

# experimental/pyproject.toml
[project]
dependencies = [
    "mai_job",
    "rocket",
    "sgl-server",
]

[tool.uv.sources]
mai_job = { workspace = true }
rocket = { workspace = true }
sgl-server = { workspace = true }

{ workspace = true } tells uv “this dependency lives in our workspace, not on PyPI.” uv resolves it to the local package. No publishing step needed — you just import it.

The dependency groups in action

Back to the root pyproject.toml. This is where dependency groups do their monorepo thing:

[dependency-groups]
dev = [
    "pytest>=8.0",
    "ruff==0.9.9",
    "pyright==1.1.408",
    "pre-commit>=4.5.0",
    # ... testing and linting tools
]
experimental = []
condor2 = []
notebook = [
    "ipykernel>=6.29.5",
    "jupyterlab>=4.4.3",
]

[tool.uv]
default-groups = ["dev", "condor2"]

Here’s how different people interact with this:

CI doesn’t sync the whole workspace — it targets specific packages. A developer running uv sync gets the dev tools. Someone running uv sync --group notebook also gets Jupyter. The default path is lean. Heavier stuff is opt-in.

Override dependencies — the escape hatch

There’s one more trick the root uses. override-dependencies:

[tool.uv]
override-dependencies = [
    # System-installed, not in uv.lock
    "flash-attn; sys_platform == 'never'",
    "triton; sys_platform == 'never'",
    # Use system torch on Linux (Docker images), uv torch on macOS
    "torch; sys_platform == 'darwin'",
    # Pin pydantic across the whole workspace
    "pydantic==2.9.2",
]

sys_platform == 'never' is a clever hack. It says “this package exists on no platform” — effectively telling uv to pretend this dependency doesn’t exist. We use it for packages that are pre-installed in our Docker images (flash-attn, triton) and shouldn’t be resolved from PyPI.

The torch; sys_platform == 'darwin' line means “only resolve torch from PyPI on macOS.” On Linux (where our training runs), torch is system-installed in the Docker image. The lockfile doesn’t need to track it there.

This section is basically the monorepo saying “I know better than the solver for these specific packages, here are the overrides.”

The PR that started this

So — the PR that confused me. A teammate wanted to add experimental, mai-layers, and marimo to the experimental dependency group:

[dependency-groups]
experimental = [
    "experimental",
    "mai-layers",
    "marimo",
]

Before this, experimental = [] — an empty group. A placeholder. The PR fills it with an actual useful set of packages: the experimental sandbox package (where people prototype), mai-layers (the core model layers — heavy, needs torch), and marimo (a modern reactive notebook that stores as .py files instead of .ipynb).

Here’s what I thought at first: “wait, does this affect everyone? Does everyone now download torch and marimo?”

No.

Because experimental is not in default-groups. Check:

default-groups = ["dev", "condor2"]

No experimental. So:

• uv sync (what everyone runs) — unchanged. Same as before.
• CI / Docker builds — unchanged. They sync specific packages, not groups.
• uv.lock — gets a bit bigger because marimo’s transitive deps get resolved. But the lockfile tracking more packages doesn’t mean they get installed.
• uv sync --group experimental — THIS is the new thing. Now you get a full experimentation environment with one command. Previously you’d have to figure out which packages to install manually.

That’s it. The whole PR is an opt-in convenience. If you don’t ask for the experimental group, you don’t know it exists. If you do ask for it, you get a curated set of packages for prototyping.

The mental model that clicked: dependency groups are like TV channels. default-groups is your basic cable. Everything else is premium — available but not on by default. Adding a channel to the lineup doesn’t change what shows up on everyone’s screen.

The cheat sheet

For when you inevitably forget all of this in two weeks:

Most of this is just vocabulary. Once you know the words, the file reads itself.