Last week I was reading a Slack thread about MFU at 8K sequence length. Something about how the distribution of sequence lengths in a dataset affects GPU utilization — packing efficiency, padding waste, the usual. People were going back and forth about whether to truncate or re-bucket, what the throughput delta looked like, whether it mattered more at 8K than 4K.
I understood maybe 30% of the words.
This is embarrassing for a specific reason: I work on the team that ships this stuff. I build the Docker images these training jobs run in. I maintain the CI that tests the code. I’m migrating the monorepo to Bazel so builds don’t take forever. Every day, I’m packaging and optimizing the infrastructure that ML engineers use to train models.
But I’ve never trained a model. Not once.
The realization
It hit me in that Slack thread. I know what a uv sync does inside a Dockerfile. I can tell you why Flash Attention takes 20 minutes to compile from source. I’ve debugged ACR DNS issues that caused image pulls to fail across geo-replicated registries at 2am. I know the build system intimately.
But ask me what a “good” loss curve looks like? What Neptune dashboards mean? How precook turns raw data into tokenized sequences? What a Rocket config actually does when it launches a training run on a GPU cluster? No idea.
I build the kitchen. I’ve never cooked a meal.
The experiment
I’m going to fix that. Starting this week, I’m spending roughly five hours per week trying a different job type from our ML training pipeline — hands-on, not just reading docs. One workload per week. Write about what I learn.
Here’s the rough list of things I want to try:
- Pretraining — the big GPU training runs. Rocket jobs on Falcon and Condor clusters. The thing the entire team exists to do.
- Fine-tuning / SFT — smaller training runs that specialize a base model on specific data.
- RLHF — the reinforcement learning pipeline. Reward models, PPO, the stuff that makes a model actually useful to talk to.
- Evaluation — running benchmarks against trained models. How do you know if your model got better?
- Data preprocessing (precook) — turning raw datasets into tokenized, packed sequences ready for training. The step before the step.
- Inference / serving — running a trained model to generate text. The other end of the pipeline.
- VLM training — multimodal. Images and text together. Different beast.
The list will probably change as I go. I’ll add things I didn’t know existed. Drop things that turn out to be boring (unlikely — so far everything looks fascinating from the outside).
What each post will cover
For each job type, I want to answer:
- What is this, actually? Not the textbook definition. What does it look like when you do it?
- How do you submit it? What configs, what tools, what commands?
- What do you watch while it runs? Where do metrics show up? What are you looking for?
- What surprised me? The gap between what I expected and what I found.
- How does this connect to my DevEx work? Because it will. Build systems and training systems have more in common than you’d think — stragglers, caching, parallelism, reproducibility. I just haven’t mapped the connections yet.
Why bother
Three reasons.
Empathy. I build infrastructure for ML engineers. When a training run fails because of a Docker image issue or a CI regression, they’re the ones who feel it. Understanding their workflow — what they’re actually doing when they submit a job, what they’re monitoring, what their day looks like — makes me a better infrastructure engineer. You can’t optimize a pipeline you don’t understand end-to-end.
Cross-pollination. I have a hunch that build systems and training systems rhyme more than people realize. Distributed builds have stragglers. Distributed training has stragglers. Docker layer caching and checkpoint resumption solve the same conceptual problem. CI flakiness and training instability are both non-determinism nightmares. I want to find those parallels, because that’s where the interesting insights live.
Accountability. Writing forces understanding. If I can’t explain what I did in a blog post, I didn’t understand it well enough. The blog is the forcing function.
The unique angle
Most ML blog posts are written by ML people. They explain things from the inside — starting from a place of deep context, filling in the theory, showing the math.
I’m coming from the outside. I know containers. I know build graphs. I know why your CI takes 40 minutes and how to make it take 12. But I don’t know what a learning rate schedule does, or why you’d pick AdamW over SGD, or what it means when someone says “the model collapsed.”
That’s the angle. A build engineer walks into a training job. What does she see?
I think there’s value in that perspective. Beginners notice things experts have stopped seeing. Infrastructure people ask different questions than research people. And honestly, I suspect a lot of engineers on ML-adjacent teams are in the same boat — close enough to see the dashboards, far enough to not know what any of it means.
What’s first
I’m starting with precook — data preprocessing. It feels like the right entry point. It’s upstream of everything else. Before you can train a model, you need data in the right format. And from what I can tell, precook is one of those things that’s deeply important but rarely explained to outsiders.
Plus, it’s probably the closest to my existing mental models. Data goes in, transformed data comes out. That’s basically what a build system does. Famous last words.
See you next week with a post full of things I got wrong about tokenization.