The LLM Benchmark That Actually Matters Is the One Nobody Publishes

MMLU, HumanEval, SWE-bench — these tell you which model is smartest. They don’t tell you which model makes you fastest.

That’s a different question entirely. And it’s the one that actually matters if you’re a power user spending 6+ hours a day with an LLM in your workflow.

The gap

Public benchmarks measure model capability in isolation. Clean prompts, controlled environments, standardized tasks. A model scores 92% on HumanEval. Great. But when I’m deep in a monorepo debugging a Bazel rule that only breaks in Docker on Linux with system Python — HumanEval tells me nothing about whether this model will waste my afternoon.

What I actually care about: my throughput with this model, on my types of work, with my prompting style. That’s a function of the model, yes, but also of my workflow, my domain, my tolerance for babysitting, and how much I trust the output.

“Which model is smartest” and “which model makes me fastest” are not the same question.

The trust tax

Here’s one that doesn’t show up in any benchmark: verification overhead.

A model that’s 90% correct but makes you read every line of output costs more wall-clock time than a model that’s 95% correct and lets you skim. The 5% accuracy gap compounds into a massive attention gap. You’re not just fixing errors — you’re looking for errors. That’s a different cognitive mode, and it’s exhausting.

This is the trust tax. It’s invisible in benchmarks because benchmarks don’t measure the human side of the loop.

What I actually want to measure

If I were building a personal benchmark, here’s what matters:

And these vary wildly by task type. A model might nail boilerplate generation but fumble multi-file refactors. Excellent at explaining code, terrible at orchestrating changes across a codebase. The aggregate score hides the distribution.

The roadmap

I’m not going to build a framework. I’m going to start small and let the structure emerge from data.

Phase 1: Shadow testing (now)

Run the same real task on 2+ models in parallel. Not synthetic benchmarks — actual work I’m already doing. Track results in a spreadsheet. Five columns: task description, model, outcome (success/partial/fail), time, notes.

Seed it with 5 tasks from this week’s actual work. Code generation, debugging, multi-file changes, research comprehension, vibe coding sessions.

Phase 2: Pattern recognition (2-3 weeks)

After a few weeks, patterns emerge. “Model X is better at multi-file orchestration.” “Model Y is faster for boilerplate but hallucinates imports.” “Opus for planning, Sonnet for implementation, Haiku for throwaway scripts.”

The tasks with the most variance between models — those are the interesting ones.

Phase 3: Formalize (when patterns stabilize)

The high-variance tasks become the benchmark suite. Known-good outputs as reference. Automated scoring where possible. Now I have a personal eval I can re-run when a new model drops, instead of vibes-testing for a week and declaring a winner.

This is basically what the Aider leaderboard does, but scoped to one person’s actual workflow instead of a standardized coding task set.

The phase model

One thing that’s already clear from informal shadow testing: different models for different phases of work.

The benchmark that matters isn’t “which model wins overall.” It’s “which model wins this phase of this type of work.” The answer is a routing table, not a single number.

I’ll report back when I have data.