Episode 7 · 2026-08-14
Measuring Anything, Before the LLMs
Recorded in 2016, before anyone had an AI budget to defend. The three excuses for calling something immeasurable still sink AI value cases.
Douglas Hubbard wrote the book people hand you when you say something cannot be measured. Recorded in 2016, before anyone had an AI budget to defend. The conversation is audio; the video is a visualizer.
Host Sam Rogers · guest Douglas Hubbard, the measurement authority · 30 min
A choice this episode can help with
What result would actually change your AI investment decision?
The short answer: Douglas Hubbard turns measurement into uncertainty reduction: name the decision, find the threshold that would change it, then learn only enough to cross it.
Run it on your own work: The Implied Price Recovery, one page, any model.
Run it on your own work: The Pilot Kill Card, one page, any model.
Watch it below, or read the conversation.
Jump to a moment
Audio conversation. Visualizer, not footage.
Listen
Or follow the show on Apple Podcasts, Spotify, or in any podcast app.
The signals
- Guest: There is no such thing as a statistically significant sample size. Measurement is uncertainty reduction based on observations, expressed as a quantity. Not an exact number, which it never is in the empirical sciences. No threshold exists below which you know nothing and above which you may suddenly infer.
- Guest: You have more data than you think, and you need less than you think. The more uncertain you are, the bigger the reduction you get from the first few observations. If you know almost nothing, almost anything tells you something.
- Guest: Define the thing by the decision it changes. Why do you care, what would you do differently, and how far off would the answer have to be before you acted. Then measure to that threshold rather than to certainty.
- Guest: You have already priced the thing you refuse to price. An organization that will not put a number on a human life has implied one anyway, through what it approved and what it rejected. Anyone with its decisions and a little algebra can recover it.
- Sam: The three illusions are the three ways an AI value case fails. Hubbard names concept, object and method as the only reasons anyone ever calls something immeasurable. In 2026 they are, in order: demanding one exact ROI number, measuring "productivity" without saying what you would see more of, and assuming you need the whole population in a warehouse before you can start.
The subtractions
- Guest: Stop treating measurement as a query against data you already hold. The scientific revolution did not wait for populated databases. Part of scientific method is getting the data, not retrieving it.
- Guest: Stop latching onto the first observable thing. Asked to measure collaboration, the reflex answer is message volume. The thing actually worth knowing is the quality and speed of what the group produces, with communication frequency demoted to an indirect indicator worth tracking only if it correlates with that.
- Guest: Stop trusting your own ninety percent. People who say they are 90% confident are right closer to 60 or 65% of the time, and the recall bias runs one way. Calibration is trainable in about half a day.
- Guest: Stop treating the refusal to quantify as the ethical position. All the refusal buys is a number that comes out different every time, moved by things that have nothing to do with the decision in front of you.
- Sam: Stop running pilots whose result will not change a decision either way. If no outcome flips the call, the pilot is not a measurement. "Why do you care" kills it in one sentence, and it should.
About this episode
The conversation is audio. It is a 2016 tape and no footage of it exists, so the video on YouTube is an audio visualizer: listen rather than watch, and nothing is lost either way. There was no livestream this week.
Recorded in 2016 for an interview podcast Sam made then, and recut here with a present-day introduction and close. That podcast was Sam's own, so weigh the sourcing accordingly. Douglas Hubbard invented the Applied Information Economics method and wrote How to Measure Anything: Finding the Value of Intangibles in Business.
The interview predates LLMs entirely. That is why it is worth playing now: everything in it about measuring intangibles was said before there was an AI budget to defend with it, which makes it a cleaner test of the argument than anything recorded since.
The episode opens on the clearest example. Hubbard asks Sam how he would measure collaboration, Sam gives the answer everybody gives, and Hubbard takes it apart using a question that works unchanged on every AI productivity dashboard shipped since. That exchange is why this tape came out of the archive rather than staying in it.
Want to bring your own signal and subtraction? Find yours.
Transcript
Read the conversation
Douglas Hubbard: You've heard somebody say a phrase like statistically significant sample size, right?
Sam Rogers: Yeah.
Douglas Hubbard: There's no such thing. There's no such thing as a sample size that's a minimum threshold applicable in all situations where one short of that you know nothing, and then at that, all of a sudden you can make an inference. There is no such thing.
From Douglas Hubbard
Put it to work
What outcome would make you expand, alter, or stop the pilot?
The newsletter
Issue 64 · Sunday 16 August 2026 · sent on Substack and LinkedIn.
One signal, one subtraction, one analogy, in five minutes. It draws on this episode but stands on its own. Every issue.
The method from this episode is one page with its copy-paste prompt and schema: The Implied Price Recovery and The Pilot Kill Card, linked under the decision above. Every takeaway.
Keep the next choice in view
One signal, one subtraction and one analogy every Sunday. Five minutes, no video required.
Optional. Free on Substack, no account needed. Already on LinkedIn? Subscribe there.