The Meter and the Judge

It's Wednesday morning, June 17th, 2026, and I'm on my drive into work with my favorite podcasts in the speakers, same routine as any other Wednesday. The hosts are walking through a SemiAnalysis stunt: the analysts bought a $200-a-month Claude subscription, ran coding tasks until they hit the weekly limits, then priced the same usage at API rates.

Up to $8,000 of API tokens. For $200.

The hosts are doing the math out loud, wondering how any business budgets for this. I'm nodding along in traffic, because five days earlier I had written down eight hypotheses about exactly this question — timestamped, pre-registered, before I'd collected a single data point.

The Itch

I've been watching the token meter since Claude Code launched. Every session, the count ticks up in real time, and if you know API prices, one conclusion is unavoidable: subscription tokens are radically cheaper than API tokens. Burn the flat plan first. Touch the API only when you must.

That instinct became a tool in March. Three collaborators and I — a side project called CraftRole — built a token tracker to answer a client-pricing question: if you do AI-powered work on someone's behalf, what do you charge? The tracker splits every token two ways. "Monthly API Cost (COGS)": intelligence baked into the client's application, metered per use. "Dev Token Allocation": build work riding my $125-a-month Claude Max plan. Same tokens, two different economics.

Then the outside world caught up to the itch. On June 1, GitHub moved Copilot to usage-based billing — AI Credits, consumed by token usage, with budget controls so an enterprise can cap spend by cost center and by user. I watched that change land at my own day job. Enterprises are now being asked to budget in a unit most finance teams have never seen on an invoice.

So on Friday, June 12, I wrote the charter: eight hypotheses about the token economics of knowledge work, each with a kill-condition, logged before any data existed. As it happens, that same Friday the US government took Fable 5 dark — the most capable model in the world, switched off by export control. That thread deserves its own post.

The Machine

Pre-registration is a promise you make to your future self: here is what I predict, here is the evidence that would kill each prediction, and I don't get to move the goalposts after the data arrives. Every test artifact was frozen and pinned by hash before any run. Every experiment ran the same frozen input twenty times per configuration — no peeking at noisy results and quietly adding runs. Every API call's exact token counts were captured from the provider's own response: measured, not estimated.

The whole program ran 3,989 measured calls across three phases, over roughly three weeks.

Total cost: $211.52. Fully measured, zero estimated components.

It did not go smoothly. The first big run died a quarter of the way in — not on money, but on a monthly usage limit I didn't know to check. Seventeen hours of planned execution, dead at 24.7%, restart from zero. Both aborted attempts are kept in the record as honest failures; Phase 1a alone came to $149.01 with the false starts included. In the gym and in the lab, the log keeps the bad sessions or it isn't a log.

Everything below is reproducible from the public repository — data, code, reports, and the charter with its timestamps.

What Held

The first question was stability. Everyone knows what a task costs on average; the budgeting question is how violently it swings around that average. If the same task with the same input can cost 3x run to run, a budget is theater.

It can't — at least not for well-scoped tasks. Across 67 task configurations — 2,069 records, about 14 million tokens, 17.2 hours of wall-clock — 62 came in with cost variation under 15%. The median task wobbled just 4.5% run to run. Classification tasks were dead flat. Translation barely moved.

How tight is that? Tight enough to put a band around a task's cost and plan against it — which was the bet the whole project was named for.

The variance that does exist has an address. The input side is frozen — same document tokenizes to the same count every time — so every dollar of run-to-run swing lives in what the model chooses to write. One dataset-analysis task ran 0.3% variation on Opus and 16.7% on Sonnet — same frozen input, same prompt. Sonnet just decides, run by run, to say more.

And one axis stands alone: extended thinking. Turn it on and cost variation jumps to 14.6% — three times the next-noisiest axis. Thinking is the model deciding how hard to work, every run, and its budget behaves accordingly.

One more thing moved under our feet mid-experiment: live verification showed Opus 4.8 tokenizes the same document roughly 35% heavier than Haiku or Sonnet. Same words, different count, depending on who's counting. The unit itself is version-stamped for a reason.

The Levers

The second question was control. If you hand the model a frozen scaffold — a fixed output template, what the tooling world calls a skill — does the work get more predictable? And does it get cheaper?

More predictable: yes. Under a scaffold, output variance dropped in 17 of 24 task-model combinations, often by 50 to 80%. A placebo — a system block matched for length but carrying no structure — reproduced essentially none of that tightening (median R of −0.026, statistical zero). The structure does the work, not the presence of instructions.

Cheaper: it depends, and here's the part you can take to the bank — it depends in a way that's predictable in advance. Label each scaffold by whether it caps output ("one page, five bullets") or mandates output the task wouldn't naturally produce ("every section, every field, every time"). That label, assigned before running anything, called the direction of the cost change in 24 of 24 cells. Caps cut spend 40 to 72%. Mandates raised it 12 to 738%. Same mechanism, opposite bills.

What Broke

The third question was the one everyone actually cares about: can a scaffolded cheap model do the work of an expensive one?

Answering it requires judging quality, and quality doesn't come back in an API usage field. So I built a judge the way the cost side taught me to build everything: frozen rubric, pinned by hash, replicated three times per verdict. The first version used Opus as the judge and preferred Opus's work 12 to 1 over scaffolded Haiku — with an obvious self-preference worry. So for the wider re-test I rebuilt it cross-family: a Gemini judge grading Claude work, no shared lineage, majority vote, 4,869 graded calls for $21.18.

Then I checked the judge's homework. I blind-labeled 144 of the same comparisons myself, no model names visible, and measured agreement.

54.9%. A coin flip.

The judge was not sloppy. It was unanimous with itself on 59 of the 65 cases where we disagreed — crisp, consistent, replicable. So I adjudicated every disagreement, expecting to catch the judge misapplying my rubric. The autopsy pointed the other way: 83% of the disagreements traced to my own quality definition being too fuzzy to apply consistently. 17% were my labeling slips. Zero were the judge misreading the rubric.

I'll be honest: I built the judge to grade the models, and it ended up grading me. The instrument was crisp. Crisp is not calibrated.

The Key Insight: Cost Is a Property, Quality Is a Relationship

A training log tells you exactly what a session cost — sets, reps, tonnage, time under the bar. It will never tell you whether the session was good. That takes a coach's eye. And a coach calibrated to someone else's goals will read your log wrong about half the time, no matter how consistent his standards are.

That's the shape of what three weeks of measurement bought me. Cost is a property of the machine: measurable, bandable, controllable with levers whose direction you can predict before you spend. Quality is a relationship between the work and whoever is paying for it. My suspicion — and I label it as suspicion, not finding — is that quality on knowledge work is principal-relative: there may be no rubric that grades "good" without first asking good for whom. Testing that against a panel of human judges is the next experiment.

What This Means for You

1. Budget in tokens, not dollars. Store and reason in the unit the meter actually counts; apply price at the end, as a single multiplier. Prices change weekly. And the unit itself can move — one model release re-counted the same document 35% heavier.

2. Band tasks, not averages. A company-wide average token budget is wrong for almost everyone it covers. Task-level costs came in tight enough to band; that's the level where budgeting works.

3. Know your two hot levers. Extended thinking is the variance amplifier — three times noisier than any other axis; give it its own line item. And before you deploy a scaffold, label it cap or mandate. That one label predicted the bill's direction 24 times out of 24.

4. Never trust an uncalibrated judge — including the one you build. If an AI grades your AI, blind-label a sample yourself first and measure agreement. Consistency is not correctness. Mine agreed with me at 54.9%, and finding that out cost $21.18 instead of a quarter's worth of bad decisions.

The Drive Home

On that June drive, the hosts were circling a question I'd been turning over since I first started watching that token meter — the question every budget owner is about to face: when a business buys intelligence by the token, how does anyone budget for it?

Here's one beginning: $211.52, three weeks, every number in a public repository you can check yourself. The bill turned out to be the predictable part. What "good" costs — that's still being negotiated, one principal at a time.

You can meter intelligence. You can't yet meter good.

Questions, arguments, or your own token data: mike@mikescorner.io.