Lessons from Batch-Migrating Seven Go Services to Bazel

2025/01/02

BUILDbazelcigo⭐

I spent a few weeks converting seven Go microservice repos to Bazel. Not one at a time, like a sane person. All at once, in parallel, bouncing between them like a browser with 40 tabs open.

Here’s what I learned. Some of it is about Bazel. Some of it is about CI. Some of it is about how my brain works.

Pick the right repos first

You’re staring at a spreadsheet of dozens of service repos. Some have been partially Bazelized. Some haven’t been touched. Some run CI 50 times a week. Some run it twice a month. You can’t do them all, and even if you could, you shouldn’t start randomly.

The prioritization criteria that actually worked:

I ended up sorting by “already Bazelized + has an SCM example to copy from” first, then by CI bottleneck severity. The first repo I converted took a day. The next three took a few hours each. Pattern recognition compounds fast.

The four-step CI conversion

Every repo followed the same pattern. Once I saw it, the whole batch became mechanical.

Step 1: Negotiate with repo owners. You’re changing their build. They need to know. This is the boring human step, but skip it and you’ll get your MR rejected by someone who’s never heard of Bazel and doesn’t appreciate surprise build system changes.

Step 2: Create a parallel build script. Don’t touch the existing build.sh. Create build.bazel.sh next to it. Same directory, different file. This means the old CI still works while you’re iterating on the new one. No risk. No “I broke prod because I was experimenting.”

Step 3: Configure CI to use the new script. In most CI systems, this is a config change — point the build job at build.bazel.sh instead of build.sh. Ideally you can A/B test: run both scripts, compare outputs, gain confidence before switching.

Step 4: Write the actual build script. This is where the real work lives. The pattern that emerged across all seven repos:

#!/bin/bash

# 1. Install bazelisk (if not present)
if [[ $(which bazel) == "" ]]; then
    curl -s "https://your-internal-tools/install.sh" | bash -s -- "bazel/tool/bazelisk"
fi

# 2. Update Go dependency repos for Bazel
bazel run //:gazelle-update-go-repos || { echo "gazelle-update-go-repos failed"; exit 1; }

# 3. Regenerate BUILD files from source
bazel run //:gazelle || { echo "gazelle failed"; exit 1; }

# 4. Build the target
bazel build //:my_service || { echo "build failed"; exit 1; }

# 5. Copy binary to output directory
mkdir -p output/bin
cp -rf bazel-bin/my_service_/my_service output/bin/${RUN_NAME}

That’s it. Five commands. Install the tool, sync deps, generate build files, build, copy binary. Every repo is a variation on this skeleton. The variables change — target name, output path, whether you need coverage tooling — but the shape is identical.

Finding the right target name? bazel query 'kind(go_binary, //...)' tells you what go_binary targets exist in the repo. The binary output path follows a predictable pattern: bazel-bin/<target>_/<target>.

The coverage tool gotcha

Some services needed code coverage instrumentation as part of CI. The existing build.sh scripts ran a Go coverage annotation tool that modifies source files in-place — injecting coverage counters into every function and branch.

Sounds simple. In practice, it’s a landmine.

The coverage tool runs go commands internally. In a Bazel build, the Go SDK lives inside Bazel’s output base, not in your system PATH. If the coverage tool picks up the system Go (say, 1.16) while Bazel is using its own managed Go SDK (say, 1.21), you get version mismatches, incompatible generated code, and builds that fail with cryptic errors about import paths.

The fix: use --run_under to prepend Bazel’s Go SDK to the PATH before the coverage tool runs.

WORKSPACE_PATH=$(bazel info workspace)
GO_SDK_PATH=$(bazel info output_base)/external/go_sdk/bin

bazel run --run_under="PATH=${GO_SDK_PATH}:$PATH " \
    //:go_coverage -- annotate --work-path ${WORKSPACE_PATH}

bazel info output_base gives you the root of Bazel’s managed external dependencies. The Go SDK lives at external/go_sdk/bin under that. By putting it first in PATH, the coverage tool uses the same Go binary that Bazel uses. No version mismatch.

The other subtlety: the annotation tool modifies .go files in-place. It literally injects coverage counter variables into your source code. After it runs, git diff shows every .go file as modified. This means:

Why it works locally but fails in CI

This came up on almost every repo. Build passes on my machine, fails in CI. Every time, it was one of these:

Toolchain version mismatch. My laptop has Go 1.21. The CI image has Go 1.16. A dependency uses a feature introduced in 1.18. Works locally, breaks in CI. The fix is either pinning the CI image or — better — letting Bazel manage the Go SDK entirely so the system Go doesn’t matter.

Missing or stale lock files. go.sum on my machine includes everything because I’ve run go mod tidy recently. The go.sum in the repo is three months old and missing entries for indirect dependencies that got added since. CI does a fresh checkout, runs go build, and fails because go.sum doesn’t match go.mod. The fix: always commit go.sum after running go mod tidy, and make CI fail loudly if they’re out of sync.

Network access differences. My laptop can reach the public Go module proxy. CI runs in a restricted network that can only reach the internal proxy. A dependency resolves differently — or not at all — through the internal proxy. The fix: configure GOPROXY and GONOSUMCHECK explicitly in the build script, don’t rely on defaults.

Cache differences. I’ve been building this repo for weeks. My local Bazel cache has everything warm. CI starts cold every time (or uses a shared cache that may or may not have the right entries). A build that takes 30 seconds locally takes 10 minutes in CI — and sometimes the cache miss reveals a dependency that’s been silently satisfied by a stale cache entry.

latest semantics. The old build script runs go get tool@latest. On Monday that pulls v1.2.3. On Wednesday it pulls v1.3.0, which has a breaking change. The build is non-deterministic by design. Bazel’s whole philosophy is eliminating this — pin everything, hash everything, never use latest.

The debugging checklist I wish I had on day one:

  1. What Go version is CI using? (go version)
  2. Is go.sum up to date? (go mod tidy && git diff go.sum)
  3. Can CI reach the module proxy? (curl -s $GOPROXY/module/@v/list)
  4. Is the build using any latest or floating version? (grep -r "latest" build.sh)
  5. What’s in the Bazel cache? (bazel info output_base to check, or bazel clean to eliminate it as a variable)

The ADHD tax on batch work

Here’s the part that’s not about Bazel.

Seven repos. Each one needs: clone, branch, write build script, test locally, test in CI, debug failures, iterate, submit MR. Multiply by seven. That’s 40+ distinct tasks across seven different contexts.

My brain does not handle this well.

I’d start on repo A, hit a CI failure, context-switch to repo B while waiting, get deep into repo B’s coverage tool issue, forget where I was on repo A, open repo C to “make progress,” realize C needs the same fix as A, go back to A, lose track of C.

The temptation — and I fell for it, repeatedly — was to stop doing the manual work and start building automation. “I should write a script that batch-clones all repos and creates branches.” “I should build a template for the build script so I can just fill in variables.” “I should create a dashboard that tracks which repos are at which stage.”

Every one of those ideas is technically correct and practically a trap. The automation takes two hours to build, saves maybe 30 minutes across seven repos, and — critically — gives my brain the dopamine hit of “building something” while the actual work sits undone.

The honest lesson: just getting the manual work done is often faster than chasing noble optimizations. Especially when the batch is small enough that automation doesn’t compound. Seven repos is not “at scale.” It’s just seven.

What actually helped:

I wrote a batch clone script anyway. It took 20 minutes and saved me maybe five:

#!/bin/bash

REPOS=(
    "https://git.example.com/team/service_a"
    "https://git.example.com/team/service_b"
    "https://git.example.com/team/service_c"
    "https://git.example.com/team/service_d"
)

BASE_DIR="$HOME/code/batch-migrate"

for REPO in "${REPOS[@]}"; do
    REPO_NAME=$(basename $REPO)
    git clone "$REPO" "$BASE_DIR/$REPO_NAME"
    cd "$BASE_DIR/$REPO_NAME"
    git checkout -b "migrate/bazelize"
    echo "Ready: $REPO_NAME"
    cd $BASE_DIR
done

Was it useful? Barely. Did writing it feel productive? Absolutely. That’s the trap.

The pattern, compressed

If I had to hand this work to someone else, here’s the cheat sheet:

  1. Prioritize: Sort repos by (CI time x CI frequency). Filter to ones already Bazelized. Start with repos that have a working example to copy.

  2. For each repo:

    • Create build.bazel.sh alongside existing build.sh
    • Pattern: install bazelisk -> gazelle-update-repos -> gazelle -> bazel build -> copy binary to output/
    • If the repo needs coverage: use --run_under with Bazel’s Go SDK path
    • If the repo has a test recording framework: handle the dirty git state from source annotation

  3. Debug CI failures with: toolchain version check, lock file freshness, network proxy config, cache state, latest usage search.

  4. Work sequentially, not in parallel. Track status in a flat list. Resist the urge to automate a seven-item batch.

The actual Bazel conversion is straightforward once you have the pattern. The hard parts are: negotiating with repo owners, debugging environment differences between local and CI, and — honestly — managing your own attention across multiple similar-but-not-identical repos.

Build systems are technical problems with human solutions. The gazelle pipeline is the easy part. The hard part is sitting down and doing the same thing seven times without getting bored and building a meta-tool.