Bazel Marker Files: How Bazel Knows When to Re-Download

Every external dependency in Bazel — every http_archive, every pip.parse package, every module from the Bazel Central Registry — gets a marker file. It’s a small file that Bazel checks on startup to decide: “Do I need to re-download this, or is the cached version still good?” Nobody talks about these files. The documentation barely mentions them. But they’re the mechanism that controls whether bazel build takes 3 seconds or 3 minutes.

Where they live

Marker files sit in the external/ directory of your Bazel output base:

ls $(bazel info output_base)/external/*.marker

@bazel_tools.marker
@rules_python+.marker
@rules_python++pip+pypi_312_click.marker
@rules_python++pip+pypi_312_numpy.marker
# ... hundreds more

One marker file per external repository. The name matches the repository name with a .marker suffix. Every time Bazel starts, it reads every single one of these files and checks whether the corresponding repository is still valid.

Three sizes of marker file

Not all marker files are equal. Their complexity scales with the type of repository they track:

Let’s look at each one.

Simple: just a hash

# @bazel_tools.marker
fda8a6529e74f9d7...

That’s it. One hash. If this hash matches what Bazel computes for the current inputs, the repository is valid. If not, re-create it. These are for repositories that Bazel generates internally — built-in tools, platform definitions. They rarely change.

Medium: hash + environment

# @rules_python+.marker
8af61aed2c5b...
ENV:PATH=/usr/local/bin:/usr/bin
ENV:HOME=/Users/shiyuanzheng

The hash covers the repository rule’s input parameters — the URL, the integrity hash, any patches. The ENV lines list environment variables that the repository rule declared as inputs via environ in its rule definition.

If you change your PATH and a repository rule depends on PATH, the marker file detects the mismatch and triggers a re-download. This is how Bazel handles repository rules that use tools from the host system — the marker tracks which environment state the cached repository was built against.

Complex: the full dependency graph

# @rules_python++pip+pypi_312_click.marker
a3f9b21e7c8d...
FILE:@@rules_python++pip+//:requirements.txt=sha256:e4b2c1d8...
FILE:@@rules_python++pip+//:BUILD.bazel=sha256:f1a2b3c4...
REPO_MAPPING:@@rules_python++pip+,rules_python,rules_python+
REPO_MAPPING:@@rules_python++pip+,bazel_tools,bazel_tools

This is where it gets interesting. Module extension-generated repositories track everything:

FILE lines: Content hashes of files that the repository rule read during creation. If requirements.txt changes, this hash changes, the marker invalidates, the repository re-generates. This is how pip.parse knows to re-resolve when you add a dependency.

REPO_MAPPING lines: Records which repository names were mapped to which actual repositories during creation. If a mapping changes — say, rules_python gets upgraded and the canonical name shifts — the marker detects it.

These markers are 4KB+ because a single pip package might depend on a dozen files and have 20+ repo mappings. Multiply by hundreds of pip packages and you have a significant chunk of I/O on every Bazel startup.

The validation algorithm

On startup, Bazel processes every marker file through the same logic:

for each .marker file:
  1. Read the file
  2. Compute current hash of the repository rule's inputs
  3. Does main hash match?  → if no, invalidate
  4. For each ENV line:
     - Read current environment variable
     - Does value match?  → if no, invalidate
  5. For each FILE line:
     - Read current file, compute sha256
     - Does hash match?  → if no, invalidate
  6. For each REPO_MAPPING line:
     - Resolve current mapping
     - Does mapping match?  → if no, invalidate
  7. All match?  → skip download, use cached repository
  8. Any mismatch?  → re-execute repository rule (download/generate)

This is conservative. Any single mismatch triggers a full re-execution of the repository rule. There’s no partial invalidation — you don’t get “re-download the tarball but keep the extracted files.” The entire repository is either valid or it’s not.

The performance implication: Bazel reads every marker file sequentially on startup. With 500 external repositories (not unusual for a large Python monorepo with many pip packages), that’s 500 file reads and potentially hundreds of file hash computations before Bazel even starts thinking about your actual build.

The two different hashes

This confused me until I traced the code. A marker file might reference two seemingly different hashes:

# In MODULE.bazel cache: fda8a652...
# In the marker file: 8af61aed...

These are not the same thing, and they’re not supposed to be.

fda8a652 is the sha256 of the MODULE.bazel file’s content. It’s stored in the module cache at ~/.cache/bazel/_bazel_user/repos/.... This tracks “has the dependency declaration changed?”

8af61aed is the hash of the repository rule’s complete input parameters — the URL, the integrity checksum, strip_prefix, patches, environment variables, everything. This is what the marker file checks. It tracks “has anything that affects the downloaded result changed?”

They’re different because they track different levels of the dependency chain. The MODULE.bazel hash tracks the declaration. The marker hash tracks the resolution. You can change MODULE.bazel in a way that doesn’t affect the resolved URL (e.g., reformatting a comment), and the marker hash stays the same. You can change the resolved URL without changing MODULE.bazel (e.g., a registry redirect), and the marker hash changes.

Four caching layers

Marker files are one piece of a four-layer caching system:

The flow:

1. Bazel reads MODULE.bazel to determine what dependencies are needed
2. For each dependency, checks the marker file against current state
3. If marker is valid → use the extracted repo in layer 4, done
4. If marker is invalid → check if tarball exists in layer 3
5. If tarball exists → re-extract (fast), update marker
6. If tarball missing → re-download (slow), extract, update marker

Layer 2 (marker files) is the decision point. It determines whether you hit layer 3 (fast path — local extraction) or need a network fetch (slow path). This is why marker file validation speed matters — it’s the gate between “instant startup” and “downloading the internet.”

Where the code lives

If you want to read the actual implementation, it’s in the Bazel Java core. Not in Starlark, not in repository rules — in the Java source of Bazel itself:

This code runs at analysis time, before any build actions execute. It’s part of Bazel’s “loading phase” — the phase where Bazel figures out what the dependency graph looks like before deciding what to build.

The fact that this is Java and not Starlark means you can’t customize it. You can’t write a repository rule that uses a different marker format or a smarter invalidation strategy. The marker mechanism is hardcoded. Repository rules only control what gets fetched — the marker validation logic is Bazel’s internal concern.

Practical implications

Why bazel clean --expunge is nuclear

bazel clean --expunge deletes the entire output base, including all marker files and extracted repositories. The next bazel build re-downloads everything. Every http_archive, every pip package, every module. In a large project, this can take 10-20 minutes just for dependency fetching.

bazel clean (without --expunge) only removes build outputs, not external repositories. The marker files survive. This is almost always what you want.

Why environment changes cause re-fetches

Switch from one terminal to another with a different PATH? If any repository rule declared environ = ["PATH"], the marker file detects the change and triggers a re-fetch. This is why Bazel sometimes re-downloads dependencies for no apparent reason — something in your environment changed that a repository rule cares about.

The fix: minimize environ usage in repository rules. Every declared environment variable is a potential invalidation trigger.

Why adding one pip package re-resolves all of them

When you add a package to requirements.txt, the FILE hash in every pip-related marker file changes (because they all depend on requirements.txt). Every pip repository invalidates. pip.parse re-runs, re-resolves the full dependency tree, re-generates all repositories.

This is correct behavior — adding a package can change the resolved versions of other packages through transitive constraints. But it means a one-line change to requirements.txt triggers a full pip re-resolution, which can take minutes in a large project.

The invisible mechanism

Marker files are infrastructure that works best when you never think about them. Fast Bazel startup? Markers validated instantly. Slow startup? Markers found mismatches and triggered re-fetches. Mysterious re-downloads? A marker detected an environment or file change you didn’t notice.

Every “why is Bazel re-downloading X” question starts with reading the marker file. The answer is always in there — which hash changed, which file was modified, which environment variable shifted. The marker file is Bazel’s debug log for dependency caching, written in a format that’s inscrutable until you know what each line means.

Now you know what each line means.