The Verification Budget

Watch the two-minute walkthrough 2:15

A two-minute walkthrough: why nobody knows how much of the work they ship was read, the three numbers, the calculator, the coverage verdict, and how to run the prompt in any model.

Play video

The decision it forces

What share of the work we ship have we actually read? Some of what goes out under our name this week was never read by a person. The question is how much, workflow by workflow, and whether that's the number we thought.

Three numbers per workflow answer it: how many outputs a week, the honest minutes to verify one properly, and the minutes that actually go on it. The rest is arithmetic.

To run it, use the calculator below, or copy the prompt into any model you already use.

The cost, in his numbers

Paul Gibbons, a consultant who builds his own agents, on Episode 10, about pull requests waiting for him after a morning upstairs:

GitHub, I don't do this anymore. But when I first started, I would just click yeah, yeah, yeah, merge, merge, merge, merge, merge, merge, merge. And and then I I would often run it without permissions because I have attention issue.

He calls himself the glue that keeps the agents together: when one finishes something, he has half an hour or an hour or two hours of work to verify, approve and revise it. With five or ten running, that's a day's work just validating, and the better part of a day, on the same episode, went on rewriting what the model wrote. A friend of his used to call the shortcut good enough for government work:

it's one of these expressions people say. actually like if you think about it hard, it's actually kind of disgusting. It means, this isn't important enough for me to devote time verifying and seeing and like improving the quality of it. there's so much wrong with that.

Nobody on the episode multiplies those out. That's what this page adds, and it's the show's arithmetic rather than Paul's: outputs times honest minutes is the verification budget, and minutes actually spent over that is coverage, the share of the work that got read.

The fields

Every field feeds the arithmetic except verifier, which is there so the minutes belong to someone. There's no worked example beyond the schematic one on the card: a filled budget is one team's confession, with names in it.

The calculator

The verdict rule

Coverage is the verdict: a percent per workflow, never an average, because a row for draft social copy and a row for customer-facing pricing shouldn't share one. No threshold is drawn, because the bar is set by what the output is for, and Paul drew it the same way: nothing stupid happened, and he's in management consulting, not nuclear weapons.

minutes_to_verify_properly is the input the whole thing rests on, and the honest figure is what it would take to catch the thing that would embarrass us, not the time it takes to glance and nod. If honest minutes make the budget impossible, the work is either not important enough to verify or too important to ship unverified, and the low coverage rows say which conversation to have first.

Run it with any model

If you would rather be interviewed than fill a table, this does the same arithmetic. It may count from a log; it may never guess the minutes.

Help me build a Verification Budget: one row per workflow or agent that hands me
finished work, so I can see what share of it actually gets read before it ships.

The arithmetic is fixed. Required minutes a week is outputs a week times honest
minutes to verify one properly. Coverage is minutes actually spent over
required, capped at 100 percent. You do the arithmetic; I supply every input.

INTERVIEW ME, three questions at a time, one workflow at a time.

1. name: the workflow or agent, in plain words.
2. outputs_per_week: how many finished things it hands me in a week. If I give
   you a merge log, approval log or ticket export, count from it and show the
   count. Otherwise ask me for my estimate and record that it is an estimate.
3. minutes_to_verify_properly: how long it takes to check one the way I would
   defend to the person it goes to. Ask me this WITHOUT suggesting a number, a
   range or a benchmark. If I say "a couple of minutes", ask me once what I
   would have to actually read or run to catch a wrong one, then record whatever
   I say.
4. verifier: the person who does the checking. A name, or null. A team name is
   null.
5. minutes_actually_spent: minutes a week that really go on it. If I answer with
   what should happen, ask for what does.

RULES YOU DO NOT BREAK
1. Never estimate minutes_to_verify_properly. It is the number the whole budget
   rests on and the one people flatter themselves about. If you fill it, the
   answer is comfortable and worthless.
2. Never combine rows into one verdict as if they were equal. Report coverage
   per row, then the totals, and say that the totals hide the difference between
   draft copy and customer-facing work.
3. Do not tell me a workflow is safe, risky, or fine. Coverage is a number; what
   it means is mine.
4. Do not ask for the contents of any output, any customer name, or any
   credential. Counts and minutes are enough.

OUTPUT
Valid JSON matching the schema verification-budget/v0.1 as published at
https://sigsub.show/takeaways/verification-budget/#the-answer-format. If you
can't read that page, use these keys, in this order: schema; workflows, a list,
each with name, outputs_per_week, minutes_to_verify_properly, verifier,
minutes_actually_spent, required_minutes and coverage_pct; totals, with
coverage_pct, hours_to_full_coverage and unverified_outputs_per_week. Then a
table sorted by coverage ascending, then one line each for: coverage across
everything, outputs shipped unverified each week, hours a week to reach full
coverage.

What to check in the answer

It may do all of the arithmetic, and count outputs from a merge or approval log. It may never estimate minutes_to_verify_properly, never combine rows into one verdict, and never call a workflow safe, risky or fine.

The part that's not optional

Run it on the workflow with the most outputs first, not the one you are proudest of. Then put a name in the verifier column for every row, and notice which rows have the same name.

{
  "schema": "verification-budget/v0.1",
  "workflows": [
    {
      "name": "",
      "outputs_per_week": 0,
      "minutes_to_verify_properly": 0,
      "verifier": null,
      "minutes_actually_spent": 0,
      "required_minutes": 0,
      "coverage_pct": 0
    }
  ],
  "totals": {
    "coverage_pct": 0,
    "hours_to_full_coverage": 0,
    "unverified_outputs_per_week": 0
  }
}