No Wrong Answers

One signal 🔭 One subtraction ➖ One analogy ☑️

Created by Sam Rogers of Snap Synapse | Episode 10 with guest Paul Gibbons | The Sunday newsletter of the weekly show at sigsub.show

A choice this issue can help with: How do you know an AI-adoption effort changed anything?

---

🔭 Signal: The overlap is maybe a dozen people

This week's guest, Paul Gibbons, has been in the change world for 30+ years and building production AI systems for the last 2, which puts him somewhere most people aren't:

• nobody on the engineering side is going to out-human-capital him
• almost nobody in the change world could out-AI him

The space in the middle, where you can talk credibly about org change and know what you're talking about with agentic AI, he reckons holds half a dozen people. Maybe a dozen. That, he says, ought to be "the coin of the realm" right now, because we're spending all the money on the technology when the hard part is the workforce part.

He turned it into a hiring rule on air: he wouldn't have anyone on his team who isn't "really good with AI", because they'll simply be slow. Then he named a colleague he admires who isn't AI-savvy and doesn't want to be. "I'm not going to work with the guy. I don't care how good he is at org change."

If you, like me, are one of those dozen or less who are great at their specific domain knowledge as well as AI, this is your time to lean in.

Your enablement program has a distribution list. The people already doing this are on a different one.

---

➖ Subtraction: Good Enough for Agentic Work

Paul runs a multi-agent setup and calls it "kind of awful", because he's the glue: every finished task costs him up to two hours to verify, and with 5-10 running concurrently he's drowning in open loops. His fix is discipline, reading the pull request even when he doesn't fully understand it, because "good enough for government work" means this wasn't important enough to verify.

For what it's worth, I think the discipline is right but the position is wrong. If you're the verification layer, the ceiling on your whole operation is your attention on your worst day, and a long queue is exactly what makes it a worst day. This is no character flaw you fix by trying harder.

So here's my subtraction: stop deciding what "good enough" means after you've seen the output. Decide it before the work is handed out, in writing, by something with no stake in it passing. Nothing grades its own homework that way, and LLMs, like people, take the "let's not and say we did" route some percentage of the time.

The diagnostic for this week: before you hand an agent the next thing, write down what would make you reject it. If you can't, you aren't ready to review the result.

Good enough is a standard or it's a shrug. The difference is whether it was written down first.

---

Watch, read, or listen

Watch the full episode: YouTube. Every format in one place: sigsub.show/episodes/ep-010. Also on Substack and LinkedIn.

Jump to a segment: the plumber · I'm the glue · no money in models · good enough for government work · the spot in the middle

---

☑️ Analogy of the Week: No Wrong Answers

Whichever box you tick, the answer is yes. Which is code for no.

There's a famous study that shall not be named showing good change management makes a project 60% more likely to succeed. Here's how it's conducted:

1. ask people how the change management was
2. ask how successful the change was
3. correlate 1 with 2

Have you ever been in a project where the change management was excellent and the project failed? Or vice versa? No, because you're asking the same question twice and calling the match evidence.

This is not unlike your AI adoption dashboard: policies published, committees formed, training completed, licenses provisioned.

"No client has ever asked me if I had evidence for what I was saying."

Call it the wrong-answer test: for any adoption metric you report, name the result that would have counted as the wrong answer. If you can't name one, the measure can't fail, and a measure that can't fail is not evidence.

Activity measures what you did. Evidence measures what changed.

---

🧰 Put It to Work

Start where the signal did: the half dozen people Paul can name are scarce because they'll hand you the wrong answer when the wrong answer is the true one. It isn't the reviewer at eleven at night with nine more in the queue, who has one answer available that gets them to bed. And it isn't the dashboard, which asks the people running the program whether the program is working.

So take the choice this episode can help with. How do you know an AI-adoption effort changed anything? Pick the number you're proudest of and say out loud what result would have made you stop. If nothing you can name would have counted as a wrong answer, you were never measuring anything. You were describing your own effort back to yourself, in a nicer font.

What AI initiative are you responsible for, and what evidence would convince you that it is working? Send me your answer, or bring the choice itself to the show.

Good luck,

Sam Rogers Testing First

---

New here? Subscribe at sigsub.show and the next episode lands in your inbox.

This week's guest: Paul Gibbons, thirty years advising boards and C-suites on the human side of technology, author of The Science of Organizational Change, Change Myths, Adopting AI and many other books.

Related reading:

• 5% Success (014): MIT put 95% of organizations at zero return from GenAI, and Tony Saldana's Why Digital Transformations Fail found nearly identical odds back in 2018, with 90% of the outcome down to change management or culture. Those are the numbers Paul's profession quotes without anyone asking where they came from.
• Declare Independence from Vanity Metrics (005): the dashboard that looks good in meetings but never changes anything. That's a form with no wrong answers on it, written before there was a study to hang it on.
• Turbulent Measures (020): drowning in vanity scores, readiness, maturity, adoption, alignment. The same inventory this issue's dashboard paragraph runs through.
• Measuring 500 Nuclear Engineers (064): Douglas Hubbard's rule that anything can be measured is the constructive half of Paul's complaint that nobody asks for evidence.