I went to BazelCon 2023 in Munich as a complete newcomer. My team at ByteDance was mid-migration to a Bazel monorepo – department-level, not company-wide yet – and I figured the fastest way to learn what problems are coming is to sit in a room with people who already have those problems.
Three days, two venues, a game night at JetBrains, and a lot of conversations about BUILD files. Here’s what stuck.
The shape of the conference
BazelCon ran October 23-25. The first day was “Community Day” – an afternoon-to-evening event co-organized by EngFlow and Tweag, hosted at Salesforce’s Munich office. Days two and three were the main conference at Google Munich.
Community Day had two parallel tracks: a beginner bootcamp (run by Billy Autrey from EngFlow) and an advanced track of lightning talks. I bounced between them. The bootcamp was solid if you were just getting started, but the lightning talks were where the interesting stuff lived.
The evening wound down at La Forchetta, sponsored by Gradle. Yes, the Gradle company sponsoring drinks at a Bazel conference. Build tool diplomacy.
Ephemeral BUILD files
The talk that got my attention most was Markus Hofbauer from Luminar: “Migration using Fully Ephemeral BUILD Files.”
The idea: don’t check BUILD files into source control. Generate them on the fly, every time.
This sounds insane until you think about the maintenance burden. BUILD files drift. Someone adds a source file, forgets to update the BUILD. Someone refactors a package, the deps list is stale. With 20,000 CI jobs – which is what Luminar had – keeping BUILD files in sync manually is a full-time job that nobody wants.
So they generate everything. Gazelle runs at the start of every build. BUILD files are .gitignore’d. The source of truth is the source code itself, not the build metadata.
My immediate questions were about CI performance and remote execution. If BUILD files don’t exist in the repo, how does remote cache keying work? How do you get reproducible builds when the generation step itself could vary? Hofbauer addressed some of this but I left wanting more details on the remote execution story.
This is the kind of approach that only works if your Gazelle extensions are rock-solid. If generation is flaky or slow, you’ve traded one maintenance burden for a worse one. But if you can pull it off, the appeal is obvious – zero BUILD file maintenance, zero merge conflicts on build metadata, and the build description is always correct by construction.
Persistent pods for fast incremental builds
Shishir Kumar from ThoughtSpot presented “Persistent Pods” – a solution to the cold-start problem in CI.
The standard CI pattern: spin up a fresh container, pull the repo, build from scratch (or from remote cache), run tests, tear everything down. Every run starts cold. Remote cache helps, but link phases and certain actions don’t cache well, and cache download time itself adds up.
ThoughtSpot’s approach: keep Kubernetes pods alive between CI runs. Attach a persistent volume for the Bazel disk cache. When the next build comes in, the pod still has a warm cache on local disk. Incremental builds are genuinely incremental, because the previous build’s outputs are right there.
It’s trading infrastructure complexity for build speed. You need to manage pod lifecycle, handle cache invalidation, deal with stale state. But the speedup for large builds is significant – you’re getting local-disk cache performance in CI, which is something remote cache fundamentally can’t match due to network latency.
I filed this one under “clever and dangerous.” The failure modes are subtle. Stale persistent volumes, pods that drift out of sync, disk pressure – all things that will bite you at 2 AM. But for teams with the infra chops to manage it, it’s a real option.
Build Server Protocol
Andrzej Gluszak from JetBrains presented the Build Server Protocol (BSP) and their bazel-bsp implementation.
The core idea borrows from LSP (Language Server Protocol). LSP made it so every editor doesn’t need a custom integration with every language – the language provides a server, the editor is a client, and they speak a common protocol. BSP does the same thing for build systems.
Build tools are servers. IDEs are clients. The protocol defines how to ask “what are the build targets?”, “compile this target,” “what are the dependencies?” Without BSP, every IDE needs a bespoke plugin for every build system. With BSP, you implement the protocol once on each side.
The bazel-bsp implementation translates BSP requests into Bazel commands under the hood. Your IDE asks “build this target” in BSP, bazel-bsp turns that into bazel build //some:target, and routes the results back.
This matters more than it sounds. IDE experience is one of the biggest friction points in Bazel adoption. When I talked to other attendees, the complaints about IDE integration came up constantly. You can have the fastest build system in the world, but if your engineers can’t get code completion or jump-to-definition working, they’ll hate it. BSP is the infrastructure that could fix that.
The rest of the lightning talks
A quick rundown of the other community day talks:
“Go in Bazel with rules_go and gazelle” (Tyler French, Uber) – Uber’s experience with Go in Bazel. rules_go and Gazelle are the gold standard for Go + Bazel, and hearing about Uber-scale usage was useful for calibrating what breaks at that size.
“Coverage with Bazel” (Ulf Adams, EngFlow) – Code coverage in Bazel is surprisingly painful. Google apparently had internal tooling for this since 2019, but the open-source story was still rough in 2023. Ulf walked through the current state and the gaps.
“Cross-invocation analytics” (Janusz Kudelka, Airbnb) – Analyzing build performance across invocations, not just within a single build. This is the “what changed between yesterday’s build and today’s build that made it 3x slower?” problem. Hard to do well, extremely valuable when you can.
“Debugging Cache Misses and Sources of Nondeterminism” (Ben Radford, Tweag) – This one ran in the beginner track but the content was anything but beginner. Cache misses are the silent killer of Bazel performance. You think you’re caching everything, then you look at the cache hit rate and it’s 60%. Tracking down why something didn’t cache is detective work, and Radford had good tooling stories.
“Buck2: optimizations and dynamic dependencies” (Neil Mitchell & Chris Hopman, Meta) – Yes, a Buck2 talk at BazelCon. Meta’s build system shares enough DNA with Bazel that the cross-pollination is real. Dynamic dependencies – where the dependency graph isn’t fully known until build time – is a problem both systems grapple with, and Buck2’s approach had some ideas worth stealing.
Game night and hallway conversations
Day one ended with a game night at the JetBrains Munich office – two S-Bahn stops from the conference venue, big green building, can’t miss it. EngFlow and JetBrains hosted, there were Bazel-themed games (yes, really), board games, IntelliJ license prizes, and enough drinks to make build systems feel fun.
The hallway conversations were arguably more valuable than the talks. Patterns I kept hearing:
- Everyone struggles with IDE integration. This was universal. The build is fast, the developer experience around it is not.
- Remote execution is aspirational for most. Google uses it. A handful of large companies use it. Everyone else is figuring out remote caching first.
- Migration is the hard part. Not learning Bazel, not configuring it – migrating an existing codebase to it. Every team had war stories.
- Gazelle is load-bearing. For any language where Gazelle works well, migration is tractable. For languages where it doesn’t, you’re hand-writing BUILD files and questioning your life choices.
What I brought home
Going in, my team was early in our migration. We’d been building department-level monorepos, getting our feet wet. The conference reframed a few things:
Ephemeral BUILD files are worth investigating. We weren’t considering this at all. After Luminar’s talk, it moved onto the “explore seriously” list. The maintenance cost of checked-in BUILD files compounds fast.
IDE experience can’t be an afterthought. We were focused on getting builds working. BSP and the general emphasis on developer experience told me we’d need to invest in auto-gazelle, editor integration, and tooling from the start – not after migration is “done.”
Cache debugging is a discipline. I’d been treating cache misses as occasional annoyances. The talks made clear that cache hit rate is a first-class metric you need to monitor and optimize continuously.
The Bazel community is smaller and more helpful than I expected. Maybe 200 people at the conference? Everyone was approachable. People shared implementation details freely. For a build tool with a reputation for complexity and a steep learning curve, the community around it is remarkably welcoming.
BazelCon 2023 was my first. I walked in knowing how to write a BUILD file and walked out understanding what the actual problems are at scale. The talks gave me vocabulary. The conversations gave me calibration. Both of those compound.