The Verification Budget

The decision it forces

Some of what goes out under our name this week was never read by a person. The question is how much, and whether that is the number we thought.

Paul Gibbons, a consultant who builds his own agents, on Episode 10, about pull requests waiting for him after a morning upstairs:

GitHub, I don't do this anymore. But when I first started, I would just click yeah, yeah, yeah, merge, merge, merge, merge, merge, merge, merge. And and then I I would often run it without permissions because I have attention issue.

And the discipline he replaced it with, which is where the phrase this page is built around comes from:

I don't claim to be an engineer, but I'm using these tools so much, I just can't give everything the pass. But it's a tendency with human beings. It's like, okay, okay, that looks pretty good. I used to have a friend that said good enough for government work.

it's one of these expressions people say. actually like if you think about it hard, it's actually kind of disgusting. It means, this isn't important enough for me to devote time verifying and seeing and like improving the quality of it. there's so much wrong with that.

The cost, in his numbers

I'm the glue that keeps all these agents together. Like when it finishes something, I have to do half an hour or an hour or two hours work to verify, approve, revise it.

And if you've got five or ten of these things running, it's kind of like awful. I've got a day's work just to kind of validate some of the stuff that my agents have done during the day. So it's not like easy money, right? I wish it was, right?

Nobody on the episode multiplies those out. That is what this page adds, and it is the show's arithmetic rather than Paul's: outputs a week, times honest minutes to verify one properly, is the verification budget. Minutes actually spent, over that, is coverage. The number that comes out is the share of the work that got read.

One hedge of Paul's governs the whole thing, and it is why the calculator works one workflow at a time:

Well, I you know, nothing stupid happened, but I'm not exactly in nuclear weapons here, right? I'm in management consulting, so

The bar is set by what the output is for. A row for draft social copy and a row for customer-facing pricing should not share a verdict, and the tool does not average them into one.

The calculator

The fields

There is no worked example on this page beyond the schematic one on its card. A filled budget is one team's confession, per workflow, with a name in the verifier column, and it is theirs.

The number people flatter themselves about

minutes_to_verify_properly is the input the whole thing rests on and the one a model must never supply. "Properly" is the question, not "at all": the honest figure is what it would take to catch the thing that would embarrass us, not the time it takes to glance and nod. Paul's own account of the gap, on the same episode:

I spent the better part of a day rewriting what Claude wrote.

Fill that column with the number you would say out loud to the person the output goes to. Then look at the coverage column and stop arguing with it.

The schema

{
  "schema": "verification-budget/v0.1",
  "workflows": [
    {
      "name": "",
      "outputs_per_week": 0,
      "minutes_to_verify_properly": 0,
      "verifier": null,
      "minutes_actually_spent": 0,
      "required_minutes": 0,
      "coverage_pct": 0
    }
  ],
  "totals": {
    "coverage_pct": 0,
    "hours_to_full_coverage": 0,
    "unverified_outputs_per_week": 0
  }
}

The Copy JSON button above emits exactly this. It is the first machine-readable output of any takeaway on this site.

Run it with any model

If you would rather be interviewed than fill a table, this does the same arithmetic. It may count from a log; it may never guess the minutes.

Help me build a Verification Budget: one row per workflow or agent that hands me finished work, so I can see what share of it actually gets read before it ships.

The arithmetic is fixed. Required minutes a week is outputs a week times honest minutes to verify one properly. Coverage is minutes actually spent over required, capped at 100 percent. You do the arithmetic; I supply every input.

INTERVIEW ME, three questions at a time, one workflow at a time.

1. name: the workflow or agent, in plain words.
2. outputs_per_week: how many finished things it hands me in a week. If I give you a merge log, approval log or ticket export, count from it and show the count. Otherwise ask me for my estimate and record that it is an estimate.
3. minutes_to_verify_properly: how long it takes to check one the way I would defend to the person it goes to. Ask me this WITHOUT suggesting a number, a range or a benchmark. If I say "a couple of minutes", ask me once what I would have to actually read or run to catch a wrong one, then record whatever I say.
4. verifier: the person who does the checking. A name, or null. A team name is null.
5. minutes_actually_spent: minutes a week that really go on it. If I answer with what should happen, ask for what does.

RULES YOU DO NOT BREAK
1. Never estimate minutes_to_verify_properly. It is the number the whole budget rests on and the one people flatter themselves about. If you fill it, the answer is comfortable and worthless.
2. Never combine rows into one verdict as if they were equal. Report coverage per row, then the totals, and say that the totals hide the difference between draft copy and customer-facing work.
3. Do not tell me a workflow is safe, risky, or fine. Coverage is a number; what it means is mine.
4. Do not ask for the contents of any output, any customer name, or any credential. Counts and minutes are enough.

OUTPUT
Valid JSON matching the schema verification-budget/v0.1 as published at https://sigsub.show/takeaways/verification-budget/, then a table sorted by coverage ascending, then one line each for: coverage across everything, outputs shipped unverified each week, hours a week to reach full coverage.

The harness that changed Sam's number

On the same episode Sam described how he verifies agent work, and it is the reason his own coverage is not what Paul's was:

being able to tether them together into a harness where they can't bluff their way out of it. Because you give anything to an agent and I can always say, Yeah, let's not do that and say we did, right? But having some evidence layer.

The shape, in his words on air: one model writes the spec and the test, a panel competes for the work, the winner does the work but cannot validate its own output, the original spec writer validates it, and a third model logs everything, all from different families. It is Harnessie, and it was named once on the episode:

Yeah, it's open source, it's free, Harnessie is the name of it.

Mentioned on air, unpaid: Harnessie is built by Sam, the show's host. No money changed hands. It is not a substitute for the budget above; it is one way to make the required minutes smaller.

What a machine may never do

It may do all of the arithmetic, and it may read a merge or approval log to count outputs. That is the part a spreadsheet does better than a person.

It may never estimate the minutes to verify one properly. That is the input the whole number rests on, and a model supplying it hands back the comfortable answer, which is the answer this page exists to take away.

The part that's not optional

Run it on the workflow with the most outputs first, not the one you are proudest of. Then put a name in the verifier column for every row, and notice which rows have the same name.

Paul's line about the expression is the standard: if the honest minutes would make the budget impossible, the work is either not important enough to verify or too important to ship unverified, and the calculator has just told us which conversation to have.