TIL: Python sys.modules Collision — Test Passes Alone, Fails Together

A test that passes every time you look at it, and fails every time you don’t. That was today’s oncall.

The symptom

PR #30118 added a new CPU test file, test_dense_model_support.py, that imports from sgl_server.utils.convert_yolo_weights_to_sgl. Somewhere in that import chain:

from sglang.srt.layers.moe.fused_moe_triton import FusedMoE

On CPU runners, sglang is installed without GPU/MoE components. And here’s the thing: sglang/srt/layers is a .py file (a module), not a directory (a package). So Python can’t traverse into sglang.srt.layers.moe:

ModuleNotFoundError: No module named 'sglang.srt.layers.moe'; 'sglang.srt.layers' is not a package

Open and shut, right? Except…

The twist: it passes in isolation

MQ CI ran test_dense_model_support.py by itself. Passed. The CI investigation bot ran it four times in a row. Passed 4/4. Bot classified it as “flaky, passes on retry.” Case closed?

No. Main Required CI ran the full sgl_server shard (259 tests) and it blew up. Every time. Deterministically.

The real root cause: sys.modules poisoning

sys.modules is Python’s global import cache. It’s a plain dict. Every import statement checks it first.

Here’s what happens during full collection:

1. Some earlier test in the shard does import sglang.srt.layers. Python loads the .py file and registers it in sys.modules as a module.
2. Later, our test’s import chain tries from sglang.srt.layers.moe.fused_moe_triton import FusedMoE. Python looks up sglang.srt.layers in sys.modules, finds it’s already loaded as a module (not a package), and refuses to look for .moe as a subpackage.
3. ModuleNotFoundError.

In isolation, step 1 never happens. No prior test poisons the cache. The import chain hits the filesystem directly, discovers layers is a .py file (not a directory), and fails with the same error… wait, no. Actually, in isolation it fails too, but only if it actually reaches that code path. The test was structured so that the problematic import was gated behind a code path that didn’t execute in the isolated run.

The point is: full collection changes the import order, and sys.modules carries state between tests. This is not a flake. It’s a deterministic failure that only manifests under a specific collection order.

Why our CI pipeline missed it

This is a known gap:

• PR CI / MQ CI uses find_changed to scope the test run. New file only? Run just that file. Passes.
• Main Required CI runs the entire shard. 259 tests. Collects everything. sys.modules fills up with prior imports. Fails.
• The investigation bot runs the failing test in isolation to check for flakiness. Passes 4/4. “Must be flaky!” Wrong.

Three layers of CI, and none of them reproduce the actual failure condition: running the test as part of its full shard.

Resolution

PR #30353 reverted the original PR. Removed the test file, removed the trigger. The underlying sglang packaging issue (.py file vs directory for layers) is an upstream problem that isn’t ours to fix.

The takeaway

sys.modules is a shared mutable dict that persists across your entire pytest session. Any test that imports a module is writing to a global cache that every subsequent test reads from. When module A registers foo.bar as a file-module, and later module B tries to import foo.bar.baz expecting a package, Python says no.

The “passes alone, fails together” pattern is almost always sys.modules or some other shared process-level state. If your investigation bot only runs tests in isolation, it will never catch this class of bug. It will confidently tell you it’s a flake.

It isn’t.