Measuring Anything, Before the LLMs

Watch and listen

The signals

• Guest: There is no such thing as a statistically significant sample size. Measurement is uncertainty reduction based on observations, expressed as a quantity. Not an exact number, which it never is in the empirical sciences. No threshold exists below which you know nothing and above which you may suddenly infer.
• Guest: You have more data than you think, and you need less than you think. The more uncertain you are, the bigger the reduction you get from the first few observations. If you know almost nothing, almost anything tells you something.
• Guest: Define the thing by the decision it changes. Why do you care, what would you do differently, and how far off would the answer have to be before you acted. Then measure to that threshold rather than to certainty.
• Guest: You have already priced the thing you refuse to price. An organization that will not put a number on a human life has implied one anyway, through what it approved and what it rejected. Anyone with its decisions and a little algebra can recover it.
• Sam: The three illusions are the three ways an AI value case fails. Hubbard names concept, object and method as the only reasons anyone ever calls something immeasurable. In 2026 they are, in order: demanding one exact ROI number, measuring "productivity" without saying what you would see more of, and assuming you need the whole population in a warehouse before you can start.

The subtractions

• Guest: Stop treating measurement as a query against data you already hold. The scientific revolution did not wait for populated databases. Part of scientific method is getting the data, not retrieving it.
• Guest: Stop latching onto the first observable thing. Asked to measure collaboration, the reflex answer is message volume. The thing actually worth knowing is the quality and speed of what the group produces, with communication frequency demoted to an indirect indicator worth tracking only if it correlates with that.
• Guest: Stop trusting your own ninety percent. People who say they are 90% confident are right closer to 60 or 65% of the time, and the recall bias runs one way. Calibration is trainable in about half a day.
• Guest: Stop treating the refusal to quantify as the ethical position. All the refusal buys is a number that comes out different every time, moved by things that have nothing to do with the decision in front of you.
• Sam: Stop running pilots whose result will not change a decision either way. If no outcome flips the call, the pilot is not a measurement. "Why do you care" kills it in one sentence, and it should.

About this episode

This episode is audio only. There is no video version.

Recorded in 2016 for an interview podcast Sam made then, and recut here with a present-day introduction and close. That podcast was Sam's own, so weigh the sourcing accordingly. Douglas Hubbard invented the Applied Information Economics method and wrote How to Measure Anything: Finding the Value of Intangibles in Business.

The interview predates LLMs entirely. That is why it is worth playing now: everything in it about measuring intangibles was said before there was an AI budget to defend with it, which makes it a cleaner test of the argument than anything recorded since.

The episode opens on the clearest example. Hubbard asks Sam how he would measure collaboration, Sam gives the answer everybody gives, and Hubbard takes it apart using a question that works unchanged on every AI productivity dashboard shipped since. That exchange is why this tape came out of the archive rather than staying in it.

Want to bring your own signal and subtraction? Find yours.