ASTs for the Impatient Python Developer

I keep running into ASTs. CI scripts that parse test files. Linters. Code generators. Bazel tooling that reads Python imports to wire up BUILD files. Every time, I nod along, copy the pattern, and move on without fully understanding what I’m looking at.

“Crafting Interpreters” by Robert Nystrom is the canonical resource. Chapter 5, “Representing Code” — that’s the AST chapter. I’ve been told this multiple times. The book is 600 pages. I’m ESL with ADHD. I do not have the patience to work through 600 pages of anything, no matter how well-written.

So this is the AST chapter I wish existed. Practical, visual, pattern-based. Not “how to build an interpreter.” Just: what IS this data structure, why does it exist, and how do I use it in Python today?

The one-sentence version

An AST is your source code turned into a tree that a program can walk, inspect, and reason about.

That’s it. Source code is text for humans. An AST is the same information structured for machines. “Abstract Syntax Tree” — abstract because the concrete syntax (parentheses, semicolons, whitespace) is stripped away, leaving only the structure.

Why text isn’t enough

These two lines are identical to grep:

pytestmark = pytest.mark.slow        # real code — an assignment
config = "pytestmark = pytest.mark.slow"  # a string — just data

Same bytes. Same pattern match. But one configures your test suite and the other is a string constant that does nothing.

Grep is a text tool. It matches patterns in bytes. It doesn’t know what Python is. It can’t tell you whether pytestmark is being assigned or merely mentioned. A parser can, because a parser understands the structure.

This is the fundamental reason ASTs exist: code has structure that text doesn’t capture.

(If you want the full story on this specific problem — grep-based CI test discovery and why we replaced it — that’s Stop Grepping Your Python Tests (Use the AST).)

The tree, visually

Take a simple line:

x = 1 + 2 * 3

Here’s the actual AST:

        Module
          |
        Assign
       /       \
   Name('x')   BinOp(+)
              /         \
          Constant(1)   BinOp(*)
                      /         \
                  Constant(2)  Constant(3)

Two things to notice.

The tree encodes operator precedence. 2 * 3 is deeper — it happens first because multiplication binds tighter. You don’t need parentheses rules. The tree IS the parse result. The structure itself captures what the precedence rules mean.

The concrete syntax is gone. No equals sign. No spaces. No newline. Just the relationships: this is an assignment, the target is x, the value is an addition, the right side of the addition is a multiplication. That’s what “abstract” means — the tree keeps the meaning, drops the formatting.

Python’s ast module — the five functions you need

import ast

# 1. Parse source code into a tree
tree = ast.parse("x = 1 + 2")

# 2. Walk every node (flat iteration, no recursion needed)
for node in ast.walk(tree):
    print(type(node).__name__)
# Module, Assign, Name, BinOp, Constant, Constant

# 3. Dump the tree (debugging)
print(ast.dump(tree, indent=2))

# 4. Get source location
# Every node has: node.lineno, node.col_offset,
#                 node.end_lineno, node.end_col_offset

# 5. NodeVisitor pattern (when you need structure, not just flat walk)
class ImportFinder(ast.NodeVisitor):
    def visit_Import(self, node):
        for alias in node.names:
            print(f"import {alias.name}")
    def visit_ImportFrom(self, node):
        print(f"from {node.module} import ...")

ImportFinder().visit(tree)

That’s it. Five things. parse, walk, dump, source locations, NodeVisitor. Everything else is details.

The difference between ast.walk() and NodeVisitor: walk gives you a flat stream of every node in the tree — great for “find all X anywhere.” NodeVisitor respects the tree structure and lets you define per-node-type behavior — better when you care about where in the tree something appears.

In practice I use ast.walk() for 90% of things.

The node types you’ll actually encounter

There are ~70 node types in Python’s AST. Don’t memorize them. Here are the ones that come up in real work.

Statements — things that DO something:

Expressions — things that PRODUCE a value:

The statement/expression split matters. When you’re looking for pytestmark = ..., you want Assign (statement) where a target is Name('pytestmark'). The grep version can’t make this distinction. The AST version gets it for free.

Three real-world patterns

Everything I’ve needed to do with ASTs falls into one of three patterns. If you internalize these, you can read (and write) most AST code you’ll encounter.

Pattern 1: “Find all X in this file”

Flat walk. The simplest pattern.

def find_imports(source: str) -> set[str]:
    """Find all top-level imported package names."""
    tree = ast.parse(source)
    imports = set()
    for node in ast.walk(tree):
        if isinstance(node, ast.Import):
            for alias in node.names:
                imports.add(alias.name.split(".")[0])
        elif isinstance(node, ast.ImportFrom):
            if node.module:
                imports.add(node.module.split(".")[0])
    return imports

Walk every node. Check the type. Extract what you need. This is the same pattern our CI uses in ci/src/ci/plan/file_scanner.py — the function _parse_top_level_imports builds the dependency graph for test discovery by doing exactly this.

Pattern 2: “Find X with specific structure”

Same walk, but with nested checks on the node’s children.

def has_pytestmark(source: str) -> bool:
    """Check if a file has a pytestmark assignment."""
    tree = ast.parse(source)
    for node in ast.walk(tree):
        if not isinstance(node, ast.Assign):
            continue
        for target in node.targets:
            if isinstance(target, ast.Name) and target.id == "pytestmark":
                return True
    return False

Find Assign nodes, check if the target is named pytestmark. This is what ci/src/ci/plan/manifest.py does in _get_file_level_decorators() to extract test markers. The isinstance chain is the whole game — you’re pattern-matching on the tree structure.

Notice how the string-in-a-string problem vanishes. config = "pytestmark = pytest.mark.slow" parses to an Assign where the value is a Constant(str). The target is Name('config'), not Name('pytestmark'). The false positive is structurally impossible.

Pattern 3: “Walk a dotted name chain”

This one’s trickier because ast.Attribute is recursive. pytest.mark.slow isn’t a single node — it’s nested:

Attribute(attr='slow')
  └── Attribute(attr='mark')
        └── Name(id='pytest')

To reconstruct the dotted string:

def get_dotted_name(node: ast.expr) -> str | None:
    """Turn an Attribute chain into 'pytest.mark.slow'."""
    if isinstance(node, ast.Name):
        return node.id
    if isinstance(node, ast.Attribute):
        parts = []
        current = node
        while isinstance(current, ast.Attribute):
            parts.append(current.attr)
            current = current.value
        if isinstance(current, ast.Name):
            parts.append(current.id)
            return ".".join(reversed(parts))
    return None

Walk down the .value chain, collect each .attr, reverse at the end. This is what _get_decorator_name() in manifest.py does to read decorators like @pytest.mark.runs_on("falcon").

Once you see this pattern you’ll recognize it everywhere. Every tool that needs to resolve dotted attribute access — linters, type checkers, code generators — does some version of this walk.

What Crafting Interpreters would teach you (that you can skip)

The book covers the full pipeline of turning text into a running program. Five phases:

1. Scanning/lexing — turning raw text into tokens. Python’s tokenize module does this. ast.parse() skips straight to the tree.
2. Parsing — turning tokens into a tree. Recursive descent, Pratt parsing, operator precedence climbing. You don’t need to know these because ast.parse() does it for you.
3. Semantic analysis — type checking, scope resolution. This is what pyright and mypy do. Heavy machinery.
4. Code generation — turning the AST back into executable code. Python’s compile() builtin does this.
5. Tree-walking interpretation — executing code by walking the AST. This is literally what CPython does.

The book is genuinely great for understanding why these phases exist and how they connect. But for using ASTs in practice — for writing CI scripts, code analyzers, little tools — you only need phase 2’s output. ast.parse() hands you the tree. You walk it. That’s the job.

If you ever do want to go deeper, Chapter 5 (“Representing Code”) and Chapter 6 (“Parsing Expressions”) are the relevant ones. You can skip the rest and still get 80% of the insight.

The “aha” moment

The real insight isn’t “AST is a tree.” It’s that every tool you already use is doing this.

• ruff parses your code into an AST, walks it looking for patterns (unused imports, style violations, potential bugs)
• pyright parses into an AST, then does type inference by walking the tree
• black parses into an AST, then pretty-prints it back — the AST is the canonical form, formatting is just rendering
• Bazel’s gazelle parses Python files to find imports, uses that to generate BUILD files
• Our CI parses test files to find markers, uses that to route tests to the right runners

When you write ast.parse() + ast.walk() + a few isinstance() checks, you’re doing the same thing these tools do. Just a smaller, targeted version. A 30-line script that finds all functions with a specific decorator is the same fundamental operation as pyright analyzing your entire codebase — just with a much narrower question.

(For the full picture of how AST-based test discovery fits into our GPU CI pipeline, see How Our GPU Tests Actually Work.)

Quick reference card

Copy-paste this. It’s all you need to start.

import ast

# Parse a file
tree = ast.parse(open("file.py").read())

# Walk all nodes (flat)
for node in ast.walk(tree):
    if isinstance(node, ast.FunctionDef):
        print(f"Found function: {node.name} at line {node.lineno}")

# Dump for debugging
print(ast.dump(tree, indent=2))

# Common isinstance checks
isinstance(node, ast.Assign)       # x = ...
isinstance(node, ast.FunctionDef)  # def x():
isinstance(node, ast.ClassDef)     # class X:
isinstance(node, ast.Import)       # import x
isinstance(node, ast.ImportFrom)   # from x import y
isinstance(node, ast.Call)         # x()
isinstance(node, ast.Attribute)    # x.y
isinstance(node, ast.Name)        # x
isinstance(node, ast.Constant)    # 42, "hello"

Three patterns:

1. “Find all X” — ast.walk() + isinstance() check
2. “Find X with structure” — same walk, add checks on .targets, .value, .args
3. “Resolve dotted name” — walk the Attribute.value chain, collect .attr, reverse

That’s the toolkit. ast.parse() gives you the tree. ast.walk() gives you the nodes. isinstance() gives you the filter. Everything else is composition.