CI Observability: Anatomy of a 25-Minute Lint Check

Our monorepo lint CI takes up to 25 minutes. For a one-line change.

I opened the workflow YAML to understand why, and what I found was a system that nobody had looked at holistically in a long time. 19 pre-commit hooks. 6 custom checks. A full-repo type checker. An 800MB PyTorch download that never runs a single line of Python. Two separate AST walks of every .py file in the repository.

This is the anatomy of that system. Not theory, not best practices – just what actually happens when you push a PR to a million-line Python monorepo.

The dependency graph

Everything starts in cimq_required_workflows.yml, which calls lint_checks.yml. That workflow has four real jobs, one zombie, and one gate:

check-lock-file (~60s)
  |
  +-- check-sync-no-tailscale (~30-60s)
  |
  +-- pre-commit-checks (~3-8 min)
  |
  +-- typechecker (~10-25 min)  <-- THE BOTTLENECK
  |
typecheck (gate -- waits for all four)

Plus pyright-warnings, which has if: false && ... on line 377. It runs on no conditions, ever. Dead code in the workflow file. Nobody removed it.

The serialization matters. check-sync-no-tailscale depends on check-lock-file, which means it can’t start until the lock file check completes – about 60 seconds of waiting for something that would have failed naturally anyway. The pre-commit job has the same unnecessary dependency.

What actually runs in pre-commit

The .pre-commit-config.yaml defines 19 hooks from 12 repositories. In CI, pyright is skipped (it gets its own dedicated job), so 18 hooks execute. Here’s the full roster:

The standard stuff:

The language tools:

That’s the part most people know about. But after those 18 hooks finish, there’s a second stage.

The custom checks nobody knows about

After pre-commit runs, the workflow does this:

- name: Run custom lint checks
  if: success() || failure()
  run: |
    uv venv --system-site-packages --python 3.12
    uv sync --package ci --frozen
    uv run --frozen python -m ci.cli pre_commit

That if: success() || failure() means it runs even if pre-commit failed. The six custom checks:

Two of these are interesting for performance reasons.

AsyncIOPreCommit walks every .py file in every workspace package, parses each one into an AST, and checks for mismatched retry decorators. DependencyImportPreCommit does the same thing – walks every .py file, parses each into an AST, and checks import statements against declared dependencies.

Two full AST parses of every Python file in the monorepo. That’s ~11,700 files parsed twice. They share the same load_workspace_packages() function to discover files, they both walk .py files under src_path for each package, and they both call ast.parse() on every file they find. But they run independently, in sequence, each building its own file list and parsing from scratch.

Neither is incremental. Change one file in one package, and both checks still walk the entire repository. The file-to-check could be narrowed to just changed packages using the same find_changed infrastructure that our test CI already uses. But today, it’s all-or-nothing.

The typechecker: where the time actually goes

Here’s the sequence for the typechecker job, the thing that takes 10-25 minutes:

1. Checkout with submodules + fetch-depth: 2
2. Connect to Tailscale VPN (for private PyPI)
3. pip install torch==2.7.0 torchvision==0.22.0 torchaudio==2.7.0 – CPU-only, ~800MB, not cached
4. sudo apt-get install g++
5. uv venv --system-site-packages --python 3.12
6. uv sync --all-packages --frozen – all 105 workspace packages
7. NODE_OPTIONS=--max-old-space-size=6144 uv run pyright --level error --threads 6

The runner: 8 cores, 32GB RAM, 300GB disk. Timeout: 25 minutes.

That PyTorch install at step 3 is wild. 800MB of CPU-only PyTorch, torchvision, and torchaudio downloaded fresh every single CI run. No actions/cache. No pre-baked runner image. Just raw pip install from the PyTorch index, every time.

And pyright never runs Python. It’s a Node.js program. So why does it need torch?

Why a JavaScript type checker needs 800MB of Python packages

Pyright does static type analysis. It reads your code, builds a type graph, and checks that types are consistent across module boundaries. It never imports or executes anything.

But it needs to resolve types. When your code says import torch and uses torch.Tensor, pyright needs to know what Tensor is – its methods, its attributes, its type signatures. That information comes from .pyi stub files or the actual package source. For torch, that means having the package installed so pyright can read its type information.

This is fundamentally different from what ruff does. Ruff is a linter: “is this code styled correctly? Any unused imports?” Per-file, fast, no cross-module analysis. Pyright is a type checker: “will this crash at runtime because you passed a str where int was expected?” It traces types across import boundaries. If you change a return type in package A, pyright catches that package B is now broken.

That distinction matters because it’s why you can’t easily narrow pyright’s scope.

Why narrowing scope is hard

With ruff, you can lint just the changed files. The lints are per-file – an unused import in foo.py doesn’t depend on what’s in bar.py.

Types flow backwards. Change the return type of a function in package A, and package C three layers downstream might break because it was relying on that type signature through package B. To catch that, you need to type-check not just A, but everything that transitively depends on A.

That’s the reverse dependency closure problem. You need the changed packages plus all their dependents, recursively. In a 105-package monorepo where many packages depend on core libraries like mai_config or mai_layers, that closure can be huge.

The pyrightconfig.json has 108 execution environments with per-package overrides and 160 exclude patterns. It’s a carefully tuned global configuration. Splitting it into per-package runs means replicating that configuration correctly for each subset, and there’s a real risk: if the incremental check misses a dependency edge, broken types land on main.

So the status quo is: run pyright on everything, every time.

Why Bazel can’t easily help here

We’re doing a Bazel migration for build and test. Natural question: can Bazel help with type checking?

Not really, for a few reasons.

Pyright is a whole-program analyzer. It wants to see the entire type graph at once. There’s no concept of “compile the types for package A, produce a type artifact, feed it to package B’s type check.” It’s not like a C compiler where you produce .o files and link them. Pyright loads everything, resolves everything, checks everything.

No rules_pyright exists. Pyright is a Node.js tool. Bazel’s Python rules don’t know about it. Someone would have to write a custom rule set that understands how to invoke pyright, manage its Node.js runtime, and feed it the right Python environment. That’s a significant piece of work for unclear benefit.

The caching model doesn’t match. In Bazel, caching works because you can define fine-grained targets with clear inputs and outputs. But pyright’s inputs are “the entire resolved type graph of all transitive dependencies,” and a single type change in a core package invalidates everything downstream. Cache hit rates would be poor for the packages that matter most.

What could work long-term is type stub extraction – something like TypeScript’s --build mode with .d.ts files, where each package produces a type summary that downstream packages consume. But pyright doesn’t have this mode. It’s not how the tool was designed.

What actually works instead

The most practical optimization is per-package pyright using existing infrastructure.

We already have find_changed – a tool that computes which packages were affected by a PR. It knows the dependency graph. Extending it to compute the reverse dependency closure is straightforward: changed packages + all their transitive dependents.

Run pyright only on that closure. For a typical PR touching one or two packages, that might be 5-15 packages instead of 105. Type checking 15 packages is dramatically faster than 105, even with the setup overhead.

The safety net: keep a full-repo pyright run as a nightly job on main. The incremental check catches 95% of issues at PR time. The nightly catches anything the incremental missed. If a type error slips through to main, the nightly flags it within hours.

The numbers, all in one place

What to do about it

Six concrete things, roughly ordered by effort-to-impact:

1. Cache PyTorch CPU. 800MB downloaded fresh every run. Use actions/cache keyed on the torch version, or pre-bake it into the runner image. Saves 2-5 minutes per run.

2. Remove the unnecessary serialization. pre-commit-checks and check-sync-no-tailscale both needs: check-lock-file, but they don’t actually use its output. They’d fail naturally if the lock file were wrong. Removing the dependency lets them start immediately. Saves ~60 seconds.

3. Per-package pyright. Use find_changed to compute the reverse dependency closure, run pyright only on affected packages. Keep full-repo pyright as a nightly safety net. Saves 50-75% of typechecker time on most PRs.

4. Deduplicate the AST walks. AsyncIOPreCommit and DependencyImportPreCommit both parse every .py file in the repo independently. Share the file discovery and parse step, run both checks on each tree. One walk instead of two.

5. Make custom checks incremental. All six custom checks scan the entire repo even for a one-file change. Most of them could be scoped to changed packages. RuffConfigPreCommit only needs to check packages whose pyproject.toml changed. PyrightExecutionEnvPreCommit only needs to run if pyrightconfig.json or workspace members changed.

6. Remove the dead pyright-warnings job. Line 377: if: false && needs.typechecker.outputs.skip_all != 'true'. It’s been dead for a while. Clean it up.

The broader point

None of this is exotic. It’s a standard accumulation: someone adds a check, someone adds another check, the checks grow, nobody looks at them as a system. Each individual hook is reasonable. Each custom check catches real bugs. The type checker is genuinely valuable.

But when you compose them all, a one-line Python change triggers 18 pre-commit hooks across 12 tool repos, 6 custom Python checks (two of which AST-parse every file in the repository), a full uv sync of 105 packages, an 800MB PyTorch download, and a whole-repo pyright analysis on 8 cores with a 25-minute timeout.

If you maintain a monorepo, you probably have a version of this. The specifics differ – maybe it’s ESLint instead of ruff, or TypeScript instead of pyright. But the pattern is the same: every check runs on everything, every time, regardless of what actually changed.

Open your lint workflow. Count the tools. Check the dependency graph. Look for the dead jobs and the redundant passes. You might be surprised.