Stop Grepping Your Python Tests (Use the AST)

There’s a genre of shell script that starts out fine and slowly becomes load-bearing infrastructure. Ours was four lines of bash that discovered which test files carry specific pytest markers for GPU CI. It worked for months. Then it didn’t.

The shell pipeline

Our GPU test workflow needs to know which test files are tagged with markers like falcon_gpu_pr or falcon_gpu_nightly. The old discovery logic:

git ls-files -z -- ':(glob)**/test_*.py' ':(glob)**/*_test.py' | \
  xargs -0 grep -lE "^\s*pytestmark\s*=" | \
  xargs grep -lE "pytest\.mark\.${MARKER}" | \
  sort -u > "$tmp"
jq -Rnc '[inputs | select(length>0)]' < "$tmp" > gpu_tests.json

Three pipes. Find test files, filter for ones with pytestmark, filter again for the specific marker. Pack into JSON. Ship it.

This is the kind of pipeline that feels clever when you write it. Four lines, no dependencies, runs in milliseconds. The kind of thing you high-five yourself for.

Where grep lies to you

Grep is a text tool. It matches patterns in bytes. It has no idea what Python is.

Problem one: string literals. Grep can’t tell the difference between these two lines:

# This is a real marker assignment
pytestmark = pytest.mark.falcon_gpu_pr

# This is a string that happens to contain the same text
training_config = "pytestmark = pytest.mark.falcon_gpu_pr"

Both match ^\s*pytestmark\s*=. Both match pytest\.mark\.falcon_gpu_pr. Grep sees identical byte patterns. Python sees an assignment and a string constant. One configures your test suite, the other is data. Grep doesn’t know and doesn’t care.

In practice this means false positives. A test file that mentions markers in a docstring or a test fixture gets picked up as if it’s marked. That test runs on GPU CI when it shouldn’t. GPU minutes are expensive. Silent false positives in CI are the worst kind of bug because nobody notices until the bill does.

Problem two: conftest.py inheritance. In pytest, markers propagate downward. If tests/gpu/conftest.py declares:

pytestmark = pytest.mark.falcon_gpu_pr

Then every test file in tests/gpu/ inherits that marker. They don’t need to declare it themselves. Grep scans each file in isolation. It has no concept of directory hierarchy or pytest’s conftest inheritance model. So it misses every test that’s marked via conftest — which, in a well-organized test suite, might be most of them.

Problem three: extending it is miserable. The team wanted to add path-based opt-in for Falcon GPU tests. In Python, that’s an if-statement. In a shell pipeline, it’s another grep or awk stage bolted onto the chain, another place for quoting bugs, another thing that breaks when someone puts a space in a directory name.

The fix: ask Python about Python

A teammate rewrote this as a 235-line Python script using the ast module. The core logic:

def _has_marker_in_pytestmark(
    tree: ast.Module, marker: str, *, prefix: bool = False
) -> bool:
    for node in ast.walk(tree):
        if not isinstance(node, ast.Assign):
            continue
        for target in node.targets:
            if not (isinstance(target, ast.Name) and target.id == "pytestmark"):
                continue
            for child in ast.walk(node.value):
                if _is_marker_ref(child, marker, prefix=prefix):
                    return True
    return False

Walk the AST. Find Assign nodes. Check if the target is named pytestmark. If so, walk the value subtree looking for the marker reference.

This is fundamentally different from grep. String literals are ast.Constant nodes — they’ll never appear as ast.Assign targets. The false positive problem vanishes not because we added a special case for it, but because parsing the actual structure makes it impossible by construction.

For conftest inheritance, the script walks parent directories:

tests/gpu/models/test_attention.py
  → check tests/gpu/models/conftest.py
  → check tests/gpu/conftest.py
  → check tests/conftest.py

If any conftest in the chain declares the marker, the test file inherits it. This mirrors what pytest actually does at runtime. Grep can’t do this because grep doesn’t know what a directory tree means.

What 235 lines buys you

The shell pipeline was 4 lines. The Python script is 235. That’s not a win on line count. But here’s what those lines get you:

• Two modes: --all scans the whole repo. --filter takes a pre-filtered candidate list from another CI tool. The shell version could only do full scans.
• Prefix matching: --prefix flag lets falcon_gpu match both falcon_gpu_pr and falcon_gpu_nightly. In the grep version, you’d need another regex stage.
• 344 lines of tests. Unit tests covering individual functions, integration tests covering the full pipeline. Try writing unit tests for a four-line shell pipeline.

That last point is the real one. The shell pipeline was untestable. Not “hard to test” — structurally untestable. Each pipe stage depends on filesystem state and the output of the previous stage. You can’t isolate the grep logic from git ls-files without mocking the filesystem. With the Python version, you parse a string into an AST and ask questions about it. Pure functions, deterministic output, trivially testable.

The workflow change is clean:

# Before
git ls-files | grep "pytestmark=" | grep "pytest.mark.X" | sort | jq

# After
uv run python find_marked_tests.py $MARKER --all --prefix

The edge cases that keep it honest

The Python version isn’t perfect. A few things it deliberately doesn’t handle:

Decorator-based markers aren’t detected. If someone writes @pytest.mark.falcon_gpu_pr on a test function instead of using module-level pytestmark, the script won’t find it. This is a design choice — the team convention is module-level markers — but it’s a potential footgun if someone doesn’t know the convention.

AugAssign is ignored. pytestmark += [pytest.mark.falcon_gpu_pr] won’t be detected because the script only looks for Assign, not AugAssign. Nobody does this in practice, but Python allows it.

Filesystem walk vs git ls-files. The old grep pipeline used git ls-files, which respects .gitignore. The new script walks the filesystem directly, which means it could pick up test files in venv/, build/, or other untracked directories. In CI this doesn’t matter (clean checkout), but locally it could be surprising.

These are known tradeoffs, documented in review. The important thing is they’re knowable — you can read the code and reason about what it does and doesn’t handle. Good luck doing that with a grep pipeline.

The general pattern

Here’s the heuristic: when your shell pipeline needs to understand the structure of the thing it’s processing, you’ve outgrown shell.

Grep understands lines. It doesn’t understand syntax trees, scope, inheritance, or type systems. For finding files that contain a string? Grep is perfect. For finding files where a specific Python variable is assigned a specific value in a non-string context with directory-based inheritance? You need a parser.

The ast module is right there in the standard library. No pip install, no build step, no external dependency. ast.parse() gives you a tree. ast.walk() lets you traverse it. The API is simple enough that a 235-line script replaces a brittle shell pipeline and adds features the shell version couldn’t have.

The “grep is good enough” argument works until it doesn’t. And in CI, the failure mode is silent: tests running when they shouldn’t, tests not running when they should, GPU minutes burning on false positives, real regressions slipping through because discovery missed a conftest. You won’t know grep was wrong until something more expensive tells you.

235 lines of Python. Testable, extensible, correct. That’s the trade.