Episode 7 · 2026-08-14

Measuring Anything, Before the LLMs

Recorded in 2016, before anyone had an AI budget to defend. The three excuses for calling something immeasurable still sink AI value cases.

Douglas Hubbard wrote the book people hand you when you say something cannot be measured. Recorded in 2016, before anyone had an AI budget to defend. The conversation is audio; the video is a visualizer.

Host Sam Rogers · guest Douglas Hubbard, the measurement authority · 30 min

A choice this episode can help with

What result would actually change your AI investment decision?

The short answer: Douglas Hubbard turns measurement into uncertainty reduction: name the decision, find the threshold that would change it, then learn only enough to cross it.

Run it on your own work: The Implied Price Recovery, one page, any model.

Run it on your own work: The Pilot Kill Card, one page, any model.

Watch it below, or read the conversation.

Jump to a moment
  1. 0:00 There is no statistically significant sample size
  2. 3:10 How would you measure collaboration?
  3. 8:38 Define it by the decision it changes
  4. 23:09 Your ninety percent is really sixty-five
  5. 26:34 How the three illusions break an AI business case

Audio conversation. Visualizer, not footage.

Listen

The signals

The subtractions

About this episode

The conversation is audio. It is a 2016 tape and no footage of it exists, so the video on YouTube is an audio visualizer: listen rather than watch, and nothing is lost either way. There was no livestream this week.

Recorded in 2016 for an interview podcast Sam made then, and recut here with a present-day introduction and close. That podcast was Sam's own, so weigh the sourcing accordingly. Douglas Hubbard invented the Applied Information Economics method and wrote How to Measure Anything: Finding the Value of Intangibles in Business.

The interview predates LLMs entirely. That is why it is worth playing now: everything in it about measuring intangibles was said before there was an AI budget to defend with it, which makes it a cleaner test of the argument than anything recorded since.

The episode opens on the clearest example. Hubbard asks Sam how he would measure collaboration, Sam gives the answer everybody gives, and Hubbard takes it apart using a question that works unchanged on every AI productivity dashboard shipped since. That exchange is why this tape came out of the archive rather than staying in it.

Want to bring your own signal and subtraction? Find yours.

Transcript

Read the conversation

0:00

Douglas Hubbard: You've heard somebody say a phrase like statistically significant sample size, right?

0:04

Sam Rogers: Yeah.

0:04

Douglas Hubbard: There's no such thing. There's no such thing as a sample size that's a minimum threshold applicable in all situations where one short of that you know nothing, and then at that, all of a sudden you can make an inference. There is no such thing.

Read the full transcript

From Douglas Hubbard

From Douglas HubbardHow to Measure Anything: the free downloads

Put it to work

What outcome would make you expand, alter, or stop the pilot?

The newsletter

Read the newsletterMeasuring 500 Nuclear Engineers

One signal, one subtraction, one analogy, in five minutes. It draws on this episode but stands on its own. Every issue.

Keep the next choice in view

One signal, one subtraction and one analogy every Sunday. Five minutes, no video required.