The Verification Budget

The decision it forces

Some of what goes out under our name this week was never read by a person. The question is how much, and whether that is the number we thought.

Paul Gibbons, a consultant who builds his own agents, on Episode 10, about pull requests waiting for him after a morning upstairs:

GitHub, I don't do this anymore. But when I first started, I would just click yeah, yeah, yeah, merge, merge, merge, merge, merge, merge, merge. And and then I I would often run it without permissions because I have attention issue.

He uses the tools too much to give everything the pass, he says, but the tendency is human: that looks pretty good. A friend of his used to call it good enough for government work, and that phrase is where this page starts:

it's one of these expressions people say. actually like if you think about it hard, it's actually kind of disgusting. It means, this isn't important enough for me to devote time verifying and seeing and like improving the quality of it. there's so much wrong with that.

To run it, use the calculator below, or copy the prompt into any model you already use.

The cost, in his numbers

Paul calls himself the glue that keeps the agents together: when one finishes something, he has half an hour or an hour or two hours of work to verify, approve and revise it. With five or ten of these things running, that's a day's work just validating what the agents did, which is why he says it isn't easy money.

Nobody on the episode multiplies those out. That is what this page adds, and it is the show's arithmetic rather than Paul's: outputs a week, times honest minutes to verify one properly, is the verification budget. Minutes actually spent, over that, is coverage. The number that comes out is the share of the work that got read.

One hedge of Paul's governs the whole thing, and it is why the calculator works one workflow at a time: nothing stupid happened, and he's in management consulting, not, as he put it, nuclear weapons. The bar is set by what the output is for. A row for draft social copy and a row for customer-facing pricing should not share a verdict, and the tool does not average them into one.

The fields

There is no worked example on this page beyond the schematic one on its card. A filled budget is one team's confession, per workflow, with a name in the verifier column, and it is theirs.

The calculator

The verdict rule

Coverage is a number, and what it means is ours; each row gets its own, never an average. Paul's line about the expression is the standard we read it by: if the honest minutes would make the budget impossible, the work is either not important enough to verify or too important to ship unverified, and the calculator has just told us which conversation to have.

Fill that column with the number you would say out loud to the person the output goes to. Then look at the coverage column and stop arguing with it.

The number people flatter themselves about

minutes_to_verify_properly is the input the whole thing rests on and the one a model must never supply. "Properly" is the question, not "at all": the honest figure is what it would take to catch the thing that would embarrass us, not the time it takes to glance and nod. Paul's own account of the gap, on the same episode, was the better part of a day spent rewriting what the model wrote.

Run it with any model

If you would rather be interviewed than fill a table, this does the same arithmetic. It may count from a log; it may never guess the minutes.

Help me build a Verification Budget: one row per workflow or agent that hands me
finished work, so I can see what share of it actually gets read before it ships.

The arithmetic is fixed. Required minutes a week is outputs a week times honest
minutes to verify one properly. Coverage is minutes actually spent over
required, capped at 100 percent. You do the arithmetic; I supply every input.

INTERVIEW ME, three questions at a time, one workflow at a time.

1. name: the workflow or agent, in plain words.
2. outputs_per_week: how many finished things it hands me in a week. If I give
   you a merge log, approval log or ticket export, count from it and show the
   count. Otherwise ask me for my estimate and record that it is an estimate.
3. minutes_to_verify_properly: how long it takes to check one the way I would
   defend to the person it goes to. Ask me this WITHOUT suggesting a number, a
   range or a benchmark. If I say "a couple of minutes", ask me once what I
   would have to actually read or run to catch a wrong one, then record whatever
   I say.
4. verifier: the person who does the checking. A name, or null. A team name is
   null.
5. minutes_actually_spent: minutes a week that really go on it. If I answer with
   what should happen, ask for what does.

RULES YOU DO NOT BREAK
1. Never estimate minutes_to_verify_properly. It is the number the whole budget
   rests on and the one people flatter themselves about. If you fill it, the
   answer is comfortable and worthless.
2. Never combine rows into one verdict as if they were equal. Report coverage
   per row, then the totals, and say that the totals hide the difference between
   draft copy and customer-facing work.
3. Do not tell me a workflow is safe, risky, or fine. Coverage is a number; what
   it means is mine.
4. Do not ask for the contents of any output, any customer name, or any
   credential. Counts and minutes are enough.

OUTPUT
Valid JSON matching the schema verification-budget/v0.1 as published at
https://sigsub.show/takeaways/verification-budget/, then a table sorted by
coverage ascending, then one line each for: coverage across everything, outputs
shipped unverified each week, hours a week to reach full coverage.

What to check in the answer

It may do all of the arithmetic, and it may read a merge or approval log to count outputs. That is the part a spreadsheet does better than a person.

It may never estimate the minutes to verify one properly. That is the input the whole number rests on, and a model supplying it hands back the comfortable answer, which is the answer this page exists to take away.

The part that's not optional

Run it on the workflow with the most outputs first, not the one you are proudest of. Then put a name in the verifier column for every row, and notice which rows have the same name.

{
  "schema": "verification-budget/v0.1",
  "workflows": [
    {
      "name": "",
      "outputs_per_week": 0,
      "minutes_to_verify_properly": 0,
      "verifier": null,
      "minutes_actually_spent": 0,
      "required_minutes": 0,
      "coverage_pct": 0
    }
  ],
  "totals": {
    "coverage_pct": 0,
    "hours_to_full_coverage": 0,
    "unverified_outputs_per_week": 0
  }
}