I went to BazelCon 2024. These are my notes — not a recap, not a summary, but the things that stuck.
Conference notes are weird. You write fragments in real time, shorthand that made sense when the speaker was standing right there, and then two weeks later you’re staring at “watch out for IOPS” wondering which talk that was from. This post is me pulling those fragments into something coherent before the context fully evaporates.
Three things came out of this conference that I actually care about: how Snowflake runs Bazel at scale, strategies for flaky tests, and a meta-lesson about how to attend conferences without drowning.
Snowflake: Build Farms at Scale
Snowflake’s talk was the one I took the most notes on, and it shows — my notes are half admiration, half questions I never got to ask.
They run BuildBarn on Kubernetes. BuildBarn is a remote execution backend for Bazel — it’s the thing that lets you farm out build and test actions to a cluster instead of running everything on your laptop or one fat CI machine. The Kubernetes part matters because it means they can autoscale workers, manage node pools, and use all the standard k8s tooling for observability and scheduling.
The big warning from their talk: watch out for IOPS. Remote execution backends are I/O bound in ways you don’t expect. Every build action reads inputs from CAS (Content Addressable Storage) and writes outputs back. At Snowflake’s scale, that’s millions of small file reads and writes per build. If your storage layer can’t handle the IOPS, your build farm becomes a bottleneck instead of an accelerator. This isn’t a CPU problem or a memory problem — it’s a “your SSDs are crying” problem.
They also talked about optimizing snapshot spacing — which, as far as I can reconstruct from my notes, is about how frequently you checkpoint the state of the build graph or cache. Too frequent and you waste I/O. Too infrequent and a node failure means re-executing a lot of work. There’s a sweet spot, and finding it requires actual measurement, not vibes.
ITG — Incremental Target Graph — got a mention. The idea is computing only the affected portion of the target graph when inputs change, instead of re-analyzing the entire build. At Snowflake’s repo size, a full graph analysis is expensive. Incremental analysis makes bazel build //... after a small change feel fast instead of “go get coffee” slow.
They also brought up their extensions index, which I think is their internal registry of Bazel extensions and rules — a way to manage the growing ecosystem of custom rules without every team reinventing the wheel.
One thing in my notes that I still find funny: “Jenkins costs?” I wrote a genuine question to myself — I’d never used Jenkins, and someone mentioned it in the context of what their build farm replaced. The answer, apparently, is that Jenkins was the “before” picture: CI machines running Jenkins jobs, each one isolated, no shared cache, no remote execution, every build starting from scratch. BuildBarn on Kubernetes was the “after.”
Flaky Tests: The Two-Lane Strategy
The flaky tests session was the most directly applicable thing at the entire conference.
The core insight is simple and I’m going to bold it because it’s the kind of thing that sounds obvious but nobody does: separate your CI into a blocking lane and a non-blocking lane.
- Blocking lane: Stable tests only. If anything fails here, the build is red, the PR is blocked, someone has to fix it. These are tests you trust — deterministic, fast, no timeout flakiness.
- Non-blocking lane: Flaky tests live here. If something fails, it’s a warning, not a gate. You still see the failure. You just don’t block the entire team’s velocity on a test that passes 97% of the time.
This isn’t “ignoring flaky tests.” It’s triaging them. A flaky test in the blocking lane is worse than useless — it teaches developers to re-run CI until it goes green, which is the exact opposite of what CI is supposed to do. A flaky test in the non-blocking lane is visible, trackable, and fixable on its own timeline.
The Mechanics
Bazel has built-in support for this. On the test target itself:
py_test(
name = "my_flaky_test",
flaky = True, # Bazel will retry on failure
...
)
flaky = True tells Bazel to retry the test up to 3 times (configurable with --flaky_test_attempts). If it passes on retry, it’s reported as “FLAKY” rather than “PASSED” or “FAILED.” This gives you signal without breaking the build.
For filtering at the runner level, there’s --test_filter and testbridge_test_only — mechanisms to select specific tests or test methods. The testbridge_test_only environment variable is Bazel’s bridge to test frameworks: it passes the filter string to pytest/JUnit/Go’s testing package so you can run a subset of tests within a target.
There was also a reference to rules_go#3618, which deals with --test_filter behavior in Go tests — apparently there are edge cases where the filter doesn’t propagate correctly.
The Taxonomy of Flakiness
The talk broke flaky tests into categories:
- Timeouts: Test is correct but slow. Flaky because sometimes CI is under load and the timeout hits. Fix: increase timeout, or better, make the test faster.
- Non-deterministic test names: Some test frameworks generate dynamic test names (parameterized tests, property-based testing). If the name changes between runs, Bazel’s caching and filtering break. Fix: make test names deterministic.
- Granularity: Test is too coarse — it tests too many things, so any one of them can fail intermittently. Fix: split into smaller, focused tests.
Nothing here was revolutionary. But hearing it organized this way — with the explicit strategy of “put flaky stuff in the non-blocking lane, then systematically move tests to the blocking lane as you fix them” — clicked in a way that reading docs doesn’t.
Cases From the Field
A few other companies presented. Quick hits:
Booking.com (Amsterdam) — they’re using Bazel. I didn’t take detailed notes on their talk, which probably means it was solid but not surprising. Enterprise Java/JVM shop migrating to Bazel is a well-trodden path.
Luminar — autonomous vehicle company, C++ and Python stack. Interesting because automotive has hard real-time constraints, so their build correctness requirements are higher than most. Bazel’s hermeticity matters more when your software goes into a car.
JVM / Remote Persistent Workers — there was a session on remote persistent workers for JVM compilation. Persistent workers keep the compiler process alive between build actions, which avoids JVM startup overhead. At scale, this is significant — JVM startup is 2-5 seconds, and if you have thousands of Java compile actions, that adds up. Remote persistent workers extend this to remote execution backends.
Gazelle and Go both had dedicated sessions. I attended but my notes are just topic headers with no content, which means I was listening rather than writing. The Gazelle session probably covered Go dependency generation, which is Gazelle’s sweet spot.
The Meta-Lesson: How to Conference
Two weeks after BazelCon, I sat down to write up my notes for the team and realized I had a problem. My notes were fragments. Half of them needed context I no longer had. Some topics (like bzlmod replacing WORKSPACE) would require me to actually learn the thing before I could write intelligently about it.
The temptation was to do deep dives on everything. Learn bzlmod. Try it. Write a comprehensive doc. But that’s a trap — you end up spending 40 hours on conference follow-up and delivering nothing because you’re still “learning.”
Here’s the strategy I landed on, and I think it’s generalizable:
1. Prioritize by relevance, not by interest. Skim all the topics. Pick the 2-3 that directly affect your current work. Everything else gets a one-line note and a link.
2. Set time boundaries. One hour per topic for initial exploration. After that hour, decide: is a deeper dive actually necessary, or is a high-level summary enough? Most of the time, it’s enough.
3. Capture insights, not mastery. The goal of conference notes is “here’s what I learned and why it matters to us.” Not “here’s a tutorial on the thing I saw.” If a topic needs a tutorial, that’s a separate project.
4. Flag for later. “This feature will require further investigation” is a valid conclusion. It’s better than spending three days investigating and missing the deadline for the things you already understand.
5. Quick hands-on tests with strict time limits. If trying something would clarify your understanding — say, setting up a small project with bzlmod — give yourself 30 minutes. Timer on. If it’s not working in 30 minutes, you’ve learned that it’s non-trivial, which is itself useful information to report.
This isn’t about being lazy. It’s about recognizing that conference follow-up has diminishing returns, and the first 20% of effort captures 80% of the value. The remaining 80% of effort is real work that should be scoped and prioritized like any other project — not smuggled in as “writing up my conference notes.”
What I’d Tell Past-Me
Take fewer, better notes. My notes from the Snowflake talk are useful. My notes from the Gazelle talk are just the word “gazelle.” One good paragraph per session beats ten bullet fragments.
Talk to people between sessions. The hallway track at BazelCon is where you learn things like “oh, we solved that IOPS problem by switching to tmpfs for the CAS” — things that never make it into slides.
And go to the flaky tests talk. Every conference has one. It’s always the most immediately useful session, and it’s always in the smallest room.