Takeaways by situation

Someone signed for the AI system as it was assessed. Then the supplier changed the model underneath it between reviews, the way a GPT-4o update turned the assistant noticeably more agreeable in April 2025 until OpenAI rolled it back. Who needs telling, and how does the person whose name is on it keep checking?

A reader's comment on Issue 074, October 2026.

1. The Default Ledger. Find which settings are the vendor's defaults rather than a choice anyone here made, and whether anyone here can change them back. Once it's live, 3hr
2. Name the Human. Put one person's name on the system, so a change from the supplier has someone to be reported to. Once it's live, 3hr
3. Human in the Loop, Honestly. Write down what that person would have to see to refuse, so a changed model has something to fail against. Once it's live, 15min
4. The Verification Budget. Count how much of its output that person still reads each week. A sign-off nobody re-checks only covers the day it was signed. While it runs, 10min

We ran an AI pilot and everyone agrees it went well. Nobody can say what it proved, what result would have counted as failure, or whether the success it reports means anything.

1. The Pilot Kill Card. Name the decision the pilot feeds and the number that would flip it. If both results lead to the same choice, it isn't measuring anything. Before you commit, 15min
2. The Implied Price Recovery. When nobody will put a value on the outcome, recover the one we've already been using from a spend we approved and one we refused. Before you commit, 10min
3. Empty Isn't an Argument. Before trusting the pilot's own success report, feed it nothing on purpose and see whether it still says it worked. While it runs, 2hr

An AI tool gave us an answer with citations: cases, studies, articles. Some of them may not exist, and some may not say what the answer claims they say. What do we check before it goes out under our name?

1. Four Checks Before It Leaves Your Desk. Open each source, read the exact passage it points to, check its date, and decide whether we'd sign it. Until all four pass, it's a lead. While it runs, 15min
2. The One-Page Test. Cut the answer down to a one-page plain-text outline and count which claims survived with no source behind them. While it runs, 15min
3. The Verification Budget. If this happens every week, count how many of these answers anyone reads in full against how many go out. While it runs, 10min

An AI system made a decision, or said something to a customer, and it was wrong. The vendor points to its terms, the team points to the vendor, and nobody's name is on it.

1. The External Speech Map. Map every place an AI speaks to the outside world for us, and count the ones speaking with nobody's name on them. Once it's live, 3hr
2. Name the Human. List everything that acts without a person deciding first, and write whose name is on it if it's wrong. Once it's live, 3hr
3. The Obligation Calendar. List the AI obligations that have a date on them and who owns each one. A planning aid, not legal advice. Once it's live, 1hr

We told an AI agent what it mustn't do, in its instructions, a rules file or the system prompt. On a long enough run it did it anyway. Do we need better instructions, or something else?

1. The Enforcement Column. List every limit the agent is supposed to keep and what actually enforces each one: a folder it can't read, a key it doesn't hold, or only a sentence. Once it's live, 15min
2. The Default Ledger. Check which of the agent's settings are the vendor's defaults that nobody here chose, and whether anyone here can change them. Once it's live, 3hr
3. The Second Hole. Before rebuilding the agent, test the cheapest thing at each end of it twice, and only rework the middle if both ends fail. While it runs, 1hr