The Pilot Kill Card

The decision it forces

Every AI pilot produces a number. A percentage, a satisfaction score, a time-saved figure. Then the number lands, everyone reads it, and the organisation carries on exactly as it would have carried on anyway.

Douglas Hubbard ends that in one sentence, and he asks it before anything gets measured:

So why do you care?

Not what will we learn. What will we do differently. He puts the same question a second way a few minutes later, and it is the one worth writing on the wall: "Are there decisions you're going to make differently?"

A pilot is a measurement only if some result would change what happens next, and most of ours cannot name one.

To run it, copy the prompt below into any model you already use.

The flip test

Before the pilot starts, write down the decision it feeds and the number at which we would do the opposite. Roll it out above X. Kill it below Y.

Hubbard's version has the maths behind it. Measuring is worth something, he says, when our uncertainty is a wide range and that range straddles a threshold a critical decision turns on. The kill rule is the show's, not his. Hubbard argues the positive case; Sam states the negative one in the episode's close: if there's no outcome that flips the call of any decision, you're not measuring anything. That is the line this card is built on, and it belongs to the show rather than to the book.

Hubbard's three illusions of measurement, concept, object and method, are the objections the card pre-empts. A measurement is a range that got narrower, not one exact number. The thing measured has to be defined by what we would see more of if we had more of it. And we need a sample and a threshold, not the whole population.

The fields

One pilot, one page, no rows.

There is no worked example on this page. A filled card would be one of our pilots, with a real threshold and a real decision attached, and publishing that would say more about us than the method does. The card is worth more blank.

The verdict rule

When action_if_above and action_if_below say the same thing, the verdict is not a measurement, in those words.

Same thing, not the same sentence. The branches rarely read identically, and the ones worth catching never do: "roll it out" against "roll it out more slowly, with extra training", or "expand to all managers" against "address the gaps, then expand to all managers". Both of those roll out. The pace differs, the wording differs, and nothing the organisation does differs.

The test that settles it is an observer six months later who can see only what we did. If they couldn't tell which branch we took, the branches were the same, and the number we were waiting on was never going to move anything. That includes a branch that is narrower, cheaper, staged or resequenced: a smaller rollout is still a rollout.

What makes this bearable in the room is that the question is about the two sentences on the card, not about anyone's competence. Nobody's judgment is on trial. We wrote both branches down before the result came in, and either they lead somewhere different or they don't.

A pilot that earns not a measurement can be cancelled this morning, and the only cost is the meeting invites.

Run it with any model

Nothing to install, no account, nothing to sign up for. Paste this into whatever chat window we already trust. It interviews rather than answers, which is the only honest mode for this card.

Help me fill in a Pilot Kill Card for a pilot I am running or about to run. The
card exists to answer one question: is this pilot a measurement at all?

The rule it tests: a pilot is a measurement only if some result would change
what we do next. If the action I would take when the result is above the
threshold is the same as the action I would take when it is below, the pilot
cannot change anything.

INTERVIEW ME. Ask at most three questions at a time. Work through the fields in
this order, and do not move on until the answer is specific enough to act on.

1. pilot: what is being trialled, in the words I would use to a colleague.
2. why_you_care: the decision that is waiting on this result. If I answer with
   something I will learn rather than something I will do, tell me that is not
   an answer yet and ask again.
3. action_if_above: what happens if the result clears the threshold. A concrete
   action, with who does it.
4. action_if_below: what happens if it does not. Same bar.
5. threshold: the number or observation that separates those two actions. Ask
   for this AFTER the two actions, never before, because the threshold only
   means anything once the branches differ.
6. measurement_plan: how I would observe it, and by when.

RULES YOU DO NOT BREAK
1. Do not invent the threshold. Do not suggest a number, a percentage or a
   benchmark, and do not offer a range for me to pick from. If I cannot name
   one, record that I could not. That is a finding, not a gap for you to fill.
2. Do not supply why_you_care. You may tell me an answer is too vague. You may
   not write a better one for me. The question is the work.
3. Compare action_if_above and action_if_below and tell me plainly whether they
   are the same. That comparison is the only judgment you are allowed to make
   here. Same means the same destination, not the same sentence. A branch that
   is narrower, slower, cheaper, staged, resequenced, or that adds a step before
   arriving, is the same action: if both branches roll it out, the pilot gated
   nothing, however different the two descriptions read. Ask yourself whether an
   observer six months from now, seeing only what the organization did, could
   tell which branch was taken. If they could not, the branches are the same.
4. If they are the same, set verdict to "not a measurement" and say so in those
   words. Do not soften it, do not add encouragement, and do not suggest how the
   pilot could be reworded so that it passes.
5. If I have not given you both actions and a threshold, set verdict to
   "unresolved". Never set it to "measurement" on partial answers.
6. Do not evaluate whether the pilot is a good idea, whether the tool is any
   good, or whether the threshold is set at a sensible level. None of that is
   this card.

OUTPUT
Valid JSON matching the schema pilot-kill-card/v0.1 as published at
https://sigsub.show/takeaways/pilot-kill-card/, then the same thing as a short
table I can paste into a doc, then one sentence saying what I should do on the
strength of the verdict.

You do not need any credentials, customer names, vendor names or internal
figures to do this, and you should not ask for any.

What to check in the answer

It may compare the two actions and tell us they match. That is the one judgment in the whole method that is safe to automate, because it is a string comparison.

It may never invent the threshold or supply why_you_care. The moment a model fills those, the card passes and means nothing: we have a completed page, an answer we didn't reach, and the same pilot running. Hubbard's whole point is that the question is the work.

The part that's not optional

Fill it before the pilot runs, not after. A kill card written when the results are already in is not a measurement plan, it's a justification, and it will agree with whatever was decided in the meantime.

And say the quiet one out loud while both branches are still empty: if we cannot name a number that flips the call, we found that out for free, before spending a quarter to learn it. Hubbard's own close was that we could measure anything and don't have to measure everything.

{
  "schema": "pilot-kill-card/v0.1",
  "pilot": "what is being trialled",
  "why_you_care": "the decision that waits on this result",
  "threshold": "the number or observation that separates the two actions",
  "action_if_above": "what you do if the result clears the threshold",
  "action_if_below": "what you do if it does not",
  "measurement_plan": "how you would observe it, and by when",
  "verdict": "measurement | not a measurement | unresolved"
}