I Opened Our pyrightconfig.json and Found 11 Ghost Environments

Our typechecker CI job takes about five minutes per PR. That’s not terrible. It’s not great either. But it was one of those things where nobody ever opened the hood – it ran, it passed or failed, and you moved on. The way you never question the water heater until it stops making hot water.

I opened the hood.

What the job actually does

Here’s the breakdown for the typechecker job, annotated with timing from a typical run:

The runner is an 8-core machine. Pyright uses 6 threads. Two cores sitting there, idle, like interns who showed up on the wrong day.

That first line, though – pip install torch. Full CPU PyTorch. Every run. ~300MB download. No pip cache. No uv cache. The rest of the job uses uv (which has caching configured). Torch gets the raw pip treatment. The reason it’s there at all: PyTorch ships its own type stubs (since ~2.0), and pyright needs them for type resolution. Reasonable. The implementation? A full cold download of a 300MB package on every single CI run. Like driving to the store for salt every time you cook.

The ghost environments

Our pyrightconfig.json has 107 execution environments. Each one scopes pyright to a specific package – its source root, its include/exclude paths, its per-package severity overrides. Standard monorepo setup.

Eleven of those environments correspond to packages that are fully excluded from type checking. lib/bus, lib/tau2, launcher, launcherv2, lib/blobfile, lib/codeshield, lib/oai_functions, caas, apps/dsviewer, apps/rollout_viewer, experimental.

Each one has an execution environment entry. Pyright dutifully creates the environment metadata, resolves the root, prepares to check… and then finds nothing to check. The include paths are empty or the package is in the top-level exclude list. The environment exists, does work, and produces zero value.

It’s like having 11 empty rooms in your house with the lights on. The electricity bill doesn’t care that nobody’s home.

The duplicate

mai_evaluator appears twice in executionEnvironments. Two separate entries, same package, different settings:

// Entry 1
{ "root": "mai_evaluator/src", "reportIncompatibleMethodOverride": "error", ... }

// Entry 2
{ "root": "mai_evaluator/src", "reportIncompatibleMethodOverride": "warning", ... }

Pyright uses the first match. The second entry is dead weight. But it’s not just dead weight – it’s misleading dead weight. Someone looked at entry 2, saw the warning setting, and thought that’s what applied. It wasn’t.

The 100-to-7 inversion

Here’s the one that made me sit back in my chair.

reportImplicitOverride – the check that flags methods overriding a parent without the @override decorator. The top-level config doesn’t set it, which means it defaults to whatever basic mode uses. Then 100 out of 107 environments override it to "none" with a comment: // fix later.

One hundred environments. Same comment. Same override.

Only 7 packages actually enforce the check: conversation, item_response_theory, fortknox, mai_prompts, posttraining_models, rocket, tool_calling.

The exception became the rule. The // fix later comment became the universal law. This should be "none" at the top level, with those 7 packages opting in. Instead, we have 100 lines of config that all say the same thing: “we’re not doing this.”

That’s not a configuration. That’s a monument to good intentions.

The FIXME graveyard

mai_layers – one of the core packages, the one that defines our transformer building blocks. Its execution environment entry has an exclude section. In that section: 65 individual source files, listed one by one, each with a FIXME comment.

{
  "root": "mai_layers/src",
  "exclude": [
    "mai_layers/src/mai_layers/modules/attention.py",  // FIXME
    "mai_layers/src/mai_layers/modules/embedding.py",  // FIXME
    "mai_layers/src/mai_layers/modules/linear.py",     // FIXME
    // ... 62 more
  ]
}

The package has 271 non-test source files. 65 of them are excluded from type checking. That’s 24% of the codebase for one of the most critical packages in the monorepo, hidden behind FIXMEs that no one is fixing. The exclude list grows. The FIXME count grows. The actual type coverage shrinks. It’s a debt register that only ever gets new entries.

The tool nobody plugged in

The repo has ci/src/ci/tools/find_changed.py. It’s a package-aware diff detector with transitive dependency analysis. You give it a commit range, it tells you which packages changed and which packages depend on the ones that changed. Proper, graph-aware change detection.

Test CI uses it. run_cpu_tests.yml and run_gpu_tests.yml both call find_changed to scope their work. When you change mai_config, test CI knows to also run tests for mai_trainer and rocket and the other 15 packages that depend on it.

The typechecker job? It doesn’t use find_changed. It has its own bash logic that does binary all-or-nothing: run pyright on everything, every time.

The infrastructure for scoped type checking already exists. It’s actively running in the same CI pipeline, one workflow file over. It’s just… not wired up to the typechecker. Like having a sprinkler system installed but never connecting the pipes to the water main.

The dead job

There’s a pyright-warnings job in the lint workflow. Its if condition starts with false && .... Permanently disabled. The job definition is still there – the name, the steps, the configuration. All present. All inert.

Dead code in application code is bad. Dead code in CI configuration is worse, because it’s noise in the place where you go to understand your pipeline. Every person who opens that workflow file has to mentally process and then discard it.

The aspect_rules_lint irony

aspect_rules_lint v2.1.0 is in MODULE.bazel. Has been for a while. It’s the Bazel linting framework – generic enough to wrap any tool and produce per-target lint actions with caching and incrementality.

We use it for Starlark formatting. buildifier on .bzl files.

The framework that could turn pyright into a cached, per-package Bazel action – giving us incremental type checking for free – was already installed. For formatting build files. Like owning a commercial kitchen and using it to reheat leftovers.

The math

Five minutes per run. Roughly 100 PRs per week. Fifty working weeks a year.

5 min x 100 PRs/week x 50 weeks = 25,000 minutes = ~417 hours/year

Four hundred and seventeen hours. That’s a full-time engineer’s annual output. A year of someone’s work, spent watching pyright check ghost environments, download uncached PyTorch, and type-check packages nobody touched.

And that’s just machine time. The human cost is 50 developers, each waiting five minutes, each losing their train of thought, each context-switching to Slack or email or existential dread while the green checkmark loads.

The quick wins

Not everything requires a redesign. Some of this is just… picking up litter.

The last one is the real prize. Change one package, type-check one package (plus its dependents). The tool exists. The data exists. It just needs plumbing.

The lesson

I’ve spent months profiling Docker builds and optimizing test selection. I never once opened the typechecker config. It ran, it was green, and I moved on. I suspect most teams have a job like this – something that works, costs more than anyone realizes, and has never been examined because “it’s fine.”

Profile your CI. Not the code – the CI itself. Read the config files. Time the steps. Count the cores. Check whether the tools you already have are actually connected.

You don’t need a new framework or a fancy caching layer. You need 30 minutes with a stopwatch and the willingness to open a config file you’ve never read.

You might find ghosts.