A thread showed up in one of our training infra channels this week. Someone found a 2% efficiency difference between two training runs on 8,000 GPUs. Same model, same config, same hardware. The only difference was the dataset.
2% sounds like nothing. But when you’re running 8,000 GPUs at roughly $2/hr each, that’s $320/hr. Multiply by weeks of training and you’re burning a house-sized pile of money on a dataset quirk nobody noticed. The team spent a few days tracking it down, and the mechanism they found is genuinely interesting — and a great example of how problems at scale aren’t just bigger versions of small-scale problems. They’re different animals entirely.
I’m a build/CI engineer, not an ML person. I don’t train models. But the debugging pattern here — tiny input difference, invisible at small scale, catastrophic at large scale — is universal. So let me try to explain what happened, from the ground up.
What’s a GPU doing during training
If you’ve never thought about AI training, here’s the 30-second version.
A language model is a giant pile of numbers (billions of them) that gets “trained” by feeding it enormous amounts of text and adjusting those numbers until the model gets good at predicting the next word. Each adjustment step involves an absurd amount of math — matrix multiplications, mostly. A single consumer GPU can do trillions of operations per second, but training a frontier model requires so many operations that you need thousands of GPUs working in parallel.
Think of it like a restaurant kitchen. One chef can make one dish at a time. If you need to serve 10,000 dishes, you get a lot of chefs, split the work up, and coordinate so they’re all working on different parts of the same menu simultaneously.
The coordination part is the hard part.
MFU: are you actually using the hardware you’re paying for
MFU stands for Model FLOPs Utilization. FLOPs are floating-point operations — basically, multiplications and additions. Your GPU has a theoretical maximum throughput: “I can do X trillion operations per second.” MFU measures what percentage of that theoretical max you actually achieve during training.
In practice, you never hit 100%. There’s overhead — moving data between GPUs, waiting for synchronization, memory bottlenecks. A good training run might achieve 40-50% MFU. A great one hits higher.
The analogy: you have a car with a 200mph top speed. On a real road with traffic, curves, and speed limits, you average 80mph. Your “speed utilization” is 40%. MFU is the same concept for GPUs.
At 8,000 GPUs, a 2% MFU improvement means 2% more useful work from the same hardware. That’s equivalent to 160 GPUs worth of free compute. Just by changing how you prepare the data.
Tokens, sequences, and how text becomes math
Before a model can learn from text, it needs to be converted into numbers. This happens through tokenization — chopping text into small pieces called tokens. Roughly, a token is a word or a word fragment. “unhappiness” might become ["un", "happiness"]. “GPU” is probably one token. The exact split depends on the tokenizer, but the point is: text goes in, a list of numbers comes out.
A sequence is a chunk of tokens that gets fed to the model at once. Training data is a massive pile of text — books, websites, code, whatever — that gets packed into fixed-size sequences. If your max sequence length is 16,384 tokens (a common setting), each sequence is at most 16,384 tokens long.
But here’s the thing: not all sequences actually fill the full 16,384. You might have a 3,000-token document followed by a 5,000-token document packed into one sequence (8,000 tokens of “real” content, padded or packed to fit the slot). The distribution of actual content lengths per sequence varies depending on the dataset.
This matters more than you’d think.
Self-attention: the part where math gets expensive
Modern language models are transformers. The key innovation in transformers is a mechanism called self-attention: every token in a sequence “looks at” every other token to understand context. The word “bank” means something different in “river bank” vs “bank account,” and self-attention is how the model figures that out — by comparing each token against all the others.
Here’s the cost. If your sequence has n tokens, every token compares to every other token. That’s n * n comparisons. Double the sequence length and you quadruple the compute. Triple it, 9x. This is what computer scientists call O(n^2) — quadratic scaling.
Sequence length → Relative attention cost
1,000 tokens → 1x
2,000 tokens → 4x
4,000 tokens → 16x
8,000 tokens → 64x
16,000 tokens → 256x
So a sequence that’s twice as long doesn’t take twice as long to process. It takes four times as long. This is the core of everything that follows.
The straggler problem
Back to our restaurant kitchen with 8,000 chefs.
In distributed training, each GPU processes a portion of the batch (a “microbatch”) independently, then they all synchronize — every GPU shares its results, they get averaged, and the model’s numbers get updated. Then the next step starts.
The critical part: everyone waits for the slowest GPU to finish before the next step can begin.
Imagine 8,000 chefs each making a dish. 7,999 of them finish in 10 minutes. One chef got assigned a souffle and needs 15 minutes. The entire kitchen is idle for 5 minutes waiting for the souffle. That one slow chef determines the throughput of the whole operation.
In training, the “souffle” is a microbatch with disproportionately long sequences. If GPU #4,738 happens to get a microbatch where the sequences are mostly 14,000 tokens long, while everyone else got microbatches averaging 6,000 tokens, GPU #4,738 takes dramatically longer — because attention cost is quadratic. It’s not 2.3x longer. It’s more like 5.4x longer. And everyone waits.
This is the straggler problem.
Why 8K GPUs is different from 1K GPUs
Here’s where it gets wild. The team tested the same two datasets at 1,000 GPUs. The MFU difference was negligible — barely measurable. At 8,000 GPUs, it was 2%.
Same datasets. Same code. Different scale. Different outcome.
Why? Statistics.
Each training step, sequences get randomly assigned to GPUs. At 1,000 GPUs, you’re drawing 1,000 microbatches from the dataset. At 8,000 GPUs, you’re drawing 8,000. The more draws you take from a distribution, the more likely you are to hit the extremes.
Think of it like rolling dice. Roll one die — you’ll probably get something in the 2-5 range. Roll 8,000 dice — you’re guaranteed to get some 1s and some 6s. The extremes always show up at scale.
For training, this means: at 8K scale, every single step is almost guaranteed to have at least one GPU that drew an unusually long set of sequences. That GPU becomes the straggler. And because everyone waits for the straggler, the whole step slows down.
The key insight from the thread: it’s not the average sequence length that matters. It’s the tail of the distribution — how many really long sequences exist, and how much longer they are than the typical ones.
The two datasets
The two datasets in question had almost identical average sequence lengths. If you plotted histograms, they’d look nearly the same. But one dataset — the mid-training one with long-context data — had a slightly fatter right tail. Just a bit more probability mass in the “very long sequence” region.
At 1K GPUs, the batch sizes were small enough that both datasets produced roughly similar straggler behavior. At 8K GPUs, that slightly fatter tail meant that the worst-case microbatch in each step was consistently worse. Not by a huge amount — but consistently, every step, thousands of steps.
2% MFU, compounded over every step of training.
The team member who investigated this built an emulation tool to actually sample the sub-sequence length distributions. The visualization made it click — one distribution (the “bad” dataset, blue line) had a subtle bump in the 12K-16K token range. The other (the “good” dataset, red line) dropped off faster. Both were capped at 16K. But that little bump in the long tail was enough.
Why not just make all sequences the same length?
You might think: just pad everything to exactly the same length. Problem solved.
Two issues. First, padding is waste. If your real content is 3,000 tokens and you pad to 16,384, you’re doing attention computation on 13,384 tokens of garbage. That’s worse than the straggler problem — you’re guaranteeing that every GPU wastes most of its compute.
Second, in practice, you do pack sequences — you concatenate shorter documents to fill the slots. The data preprocessing step (called “precook” at the team) shuffles documents, tokenizes them, packs them greedily, and shuffles again. But greedy packing doesn’t optimize for balanced compute. It optimizes for minimizing wasted space, which is a different thing.
The solution direction: FLOPS-aware packing
This is where the thread got really interesting. Several people converged on the same insight: the right thing to balance isn’t sequence length. It’s sequence cost.
A microbatch with three 5,000-token sequences doesn’t cost the same as a microbatch with one 15,000-token sequence, even though both total 15,000 tokens. The single long sequence costs 9x the attention compute of any one of the short sequences.
The idea: instead of packing sequences to minimize padding, pack them to balance the sum of (sequence_length^2) across microbatches. This is “FLOPS-aware packing” — you’re not trying to make microbatches the same size in tokens, you’re trying to make them the same size in compute.
One of the team members working on vision-language models (where this problem is even worse because image tokens create huge variance in sequence length) confirmed that using FLOPS as the workload estimate for packing gives better MFU than using token count. They’re implementing a scheme where a 32K text sequence plus a 4K vision sequence get packed together, balanced by actual compute cost rather than raw length.
It’s a bin-packing problem. Instead of bins of equal volume, you want bins of equal weight, where “weight” is the sum of (chunk_length^2) for all chunks in the bin. This is NP-hard in general, but good approximation algorithms exist, and even a rough heuristic would beat random assignment.
What I find fascinating about this
I’m used to thinking about performance at the CI level — Docker builds, test parallelism, cache hit rates. Different domain, same shape of problem: you’re only as fast as your slowest worker, and the tail of the distribution kills you.
But the scale amplification here is something else. Two distributions that look identical to the human eye produce a measurable efficiency gap — but only at 8,000 GPUs. At 1,000 GPUs, you can’t see it. The problem literally doesn’t exist at small scale. It emerges from scale like temperature emerges from molecular motion — it’s a statistical phenomenon that only materializes when you have enough samples.
There’s a lesson in there that goes beyond ML training. When you 8x your system — your fleet, your test suite, your deployment pipeline, whatever — the problems you get aren’t 8x the old problems. They’re new problems. The tail starts to dominate. The thing that was a rounding error becomes the bottleneck.
In this case, the fix isn’t more hardware, or a faster model, or a better optimizer. It’s looking more carefully at how you prepare the data. A slightly smarter packing algorithm. A histogram that someone actually plotted.
The most expensive problems are always the ones that look like nothing.