The Pilot Kill Card

The question

Every AI pilot produces a number. A percentage, a satisfaction score, a time-saved figure. Then the number lands, everyone reads it, and the organisation carries on exactly as it would have carried on anyway.

Douglas Hubbard ends that in one sentence, and he asks it before anything gets measured:

So why do you care?

Not what will we learn. What will we do differently. He puts the same question a second way a few minutes later, and it is the one worth writing on the wall: "Are there decisions you're going to make differently?"

A pilot is a measurement only if some result would change what happens next, and most of ours cannot name one.

The flip test

Before the pilot starts, write down the decision it feeds and the number at which we would do the opposite. Roll it out above X. Kill it below Y.

Hubbard's version has the maths behind it. Measuring is worth something when our uncertainty straddles a threshold a real decision turns on:

if you've got this big wide range, and this range is straddling a threshold for a critical decision ... there's a value to measuring.

His sentence is unfinished as spoken, and it is quoted here the way he said it.

The kill rule is the show's, not his. Hubbard argues the positive case; Sam states the negative one in the episode's close: if there's no outcome that flips the call of any decision, you're not measuring anything. That is the line this card is built on, and it belongs to the show rather than to the book.

The three illusions

These are Hubbard's, and they are on the tape. They pre-empt the three objections this card gets.

I call them concept, object, and method, or you can think of dot com as a mnemonic if you like.

Concept is the belief that a measurement is one exact number. It never is. His own definition: "Measurement is uncertainty reduction based on observations expressed as a quantity." A measurement is a range that got narrower. Hold out for a single true figure for what AI returned and we will take the first confident number somebody hands us.

Object is never defining the thing being measured. Productivity. Collaboration. Enablement. Hubbard's response, verbatim: "If somebody says, 'I wanna measure collaboration,' we ask them, 'Why? And what do you mean by it? And what do you see when you see more of it?'" Ask what we would see more of if we had more of it, and the room goes quiet.

Method is assuming we need the whole population before we are allowed to begin. We don't. We need a sample and a threshold.

The five fields

One pilot, one page, no rows.

There is no worked example on this page. A filled card would be one of our pilots, with a real threshold and a real decision attached, and publishing that would say more about us than the method does. The card is worth more blank.

The verdict rule

When action_if_above and action_if_below say the same thing, the verdict is not a measurement, in those words.

It is arithmetic rather than opinion: two identical strings. Nobody has to be talked into it, nobody's judgment is on trial, and there is no version of the conversation where the pilot's sponsor gets to argue the verdict was unfair. The card either matches or it doesn't.

A pilot that earns not a measurement can be cancelled this morning, and the only cost is the meeting invites.

The schema

The page is the table. This is what the table is made of, so a script, a model or a spreadsheet can carry the same card without anyone learning a new format.

{
  "schema": "pilot-kill-card/v0.1",
  "pilot": "what is being trialled",
  "why_you_care": "the decision that waits on this result",
  "threshold": "the number or observation that separates the two actions",
  "action_if_above": "what you do if the result clears the threshold",
  "action_if_below": "what you do if it does not",
  "measurement_plan": "how you would observe it, and by when",
  "verdict": "measurement | not a measurement | unresolved"
}

verdict is derived rather than entered. It is not a measurement exactly when the two action fields match.

Run it with any model

Nothing to install, no account, nothing to sign up for. Paste this into whatever chat window we already trust. It interviews rather than answers, which is the only honest mode for this card.

Help me fill in a Pilot Kill Card for a pilot I am running or about to run. The card exists to answer one question: is this pilot a measurement at all?

The rule it tests: a pilot is a measurement only if some result would change what we do next. If the action I would take when the result is above the threshold is the same as the action I would take when it is below, the pilot cannot change anything.

INTERVIEW ME. Ask at most three questions at a time. Work through the fields in this order, and do not move on until the answer is specific enough to act on.

1. pilot: what is being trialled, in the words I would use to a colleague.
2. why_you_care: the decision that is waiting on this result. If I answer with something I will learn rather than something I will do, tell me that is not an answer yet and ask again.
3. action_if_above: what happens if the result clears the threshold. A concrete action, with who does it.
4. action_if_below: what happens if it does not. Same bar.
5. threshold: the number or observation that separates those two actions. Ask for this AFTER the two actions, never before, because the threshold only means anything once the branches differ.
6. measurement_plan: how I would observe it, and by when.

RULES YOU DO NOT BREAK
1. Do not invent the threshold. Do not suggest a number, a percentage or a benchmark, and do not offer a range for me to pick from. If I cannot name one, record that I could not. That is a finding, not a gap for you to fill.
2. Do not supply why_you_care. You may tell me an answer is too vague. You may not write a better one for me. The question is the work.
3. Compare action_if_above and action_if_below and tell me plainly whether they are the same. That comparison is the only judgment you are allowed to make here.
4. If they are the same, set verdict to "not a measurement" and say so in those words. Do not soften it, do not add encouragement, and do not suggest how the pilot could be reworded so that it passes.
5. If I have not given you both actions and a threshold, set verdict to "unresolved". Never set it to "measurement" on partial answers.
6. Do not evaluate whether the pilot is a good idea, whether the tool is any good, or whether the threshold is set at a sensible level. None of that is this card.

OUTPUT
Valid JSON matching the schema pilot-kill-card/v0.1 as published at https://sigsub.show/takeaways/pilot-kill-card/, then the same thing as a short table I can paste into a doc, then one sentence saying what I should do on the strength of the verdict.

You do not need any credentials, customer names, vendor names or internal figures to do this, and you should not ask for any.

What a machine may never do

It may compare the two actions and tell us they match. That is the one judgment in the whole method that is safe to automate, because it is a string comparison.

It may never invent the threshold or supply why_you_care. The moment a model fills those, the card passes and means nothing: we have a completed page, an answer we didn't reach, and the same pilot running. Hubbard's whole point is that the question is the work.

The part that's not optional

Fill it before the pilot runs, not after. A kill card written when the results are already in is not a measurement plan, it's a justification, and it will agree with whatever was decided in the meantime.

And say the quiet one out loud while both branches are still empty: if we cannot name a number that flips the call, we found that out for free, before spending a quarter to learn it.

'Cause you could measure anything. You don't have to measure everything, obviously.