Why repeating an experiment is often impossible, and what replaces it

0:00You already own a definition of probability, and it is a good one. Do the experiment a lot of times. Count how many of those times the thing you care about happened. Divide. Toss a coin a thousand times, get five hundred and one heads, and the probability of a head is five hundred and one thousandths. That is an honest number and there is nothing wrong with it. It is the definition you were given the first time anybody said the word probability to you, and this video is not about replacing it.

0:37It is about the one question you can ask of it that it cannot answer. But start by looking hard at the two counts inside it, because everything that follows comes from what those two numbers are. The bottom number is how many times you ran the experiment. The top number is how many of those runs came out the way you were asking about. Both of them are records of things that have already happened.

1:06Which means the number cannot exist until the experiment has been performed, and performed a great many times. No trials, no numerator. No trials, no denominator either. It is a report, and a report needs something to report on. For a coin that is no trouble at all. You can sit and toss one for an afternoon. For a die, the same. Roll it a few thousand times and the six fractions settle down where you would expect.

1:39This is the comfortable case, and it is worth naming as comfortable before we leave it. Here is what settling down actually looks like. One run of a thousand tosses, drawn as it happens. The line is the share of tosses so far that came up heads. This is not data from anywhere. It is a run this video generated, and you could generate a different one. At the start it lurches. After the first few tosses it is nowhere near a half, because a few tosses cannot be near anything.

2:16By ten it is at four tenths. By a hundred it is at forty-five hundredths. By five hundred it is on two hundred and forty-three, and by a thousand it has five hundred and one heads. Five hundred and one thousandths. One thousandth away from a half. The line is calming down, and it never stops wiggling. Both of those are important, and we will come back to the second one, because that is where the honesty lives.

2:49Now take the same definition somewhere it cannot go. What is the probability that a satellite launch fails? The definition tells you exactly what to do. Launch a lot of satellites, count the failures, divide. A launch costs somewhere in the tens of millions. So the recipe is not wrong. It is unaffordable, and unaffordable over and over again. You are not going to be handed a thousand launches to tally, and a handful of launches gives you the lurching left-hand end of that picture rather than the settled right-hand end.

3:28Which is worse than having no number, because a number from four trials looks exactly like a number from four thousand. That is the first obstacle, and it is the softer of the two, because money is the kind of thing that can in principle be found. Here is the harder one. What is the probability that a tall building is destroyed in an earthquake? Same recipe. Run the earthquake many times and count.

3:59And now no amount of money helps, because nobody can stage an earthquake. The experiment is not expensive. It is not ours to run. That is a different kind of wall. The first one says you cannot afford the trials. This one says there are no trials to be had. And notice what that does to the definition we started with. It does not make it wrong. It makes it silent.

4:27A definition that needs trials has nothing whatsoever to say about a question where trials do not exist. So the subject makes a move, and it is worth seeing it as a move rather than as a new chapter heading. It stops measuring, and starts assuming. Instead of running the experiment, we say something in advance about how the experiment behaves. And from that statement, a number falls out with no trials at all.

4:58Not one toss. Not one launch. That is a real trade and it deserves to be written down as one, with both columns filled in. What you get is a number for questions you could never have measured, and you get it immediately. What you pay is that the number now rests on the statement you made in advance, rather than on anything that happened. Which statement? The one from the previous topic, and it is the only one being bought here.

5:31That the outcomes, as you have named them, are equally likely. Name the outcomes so that nothing separates them, and then the chance of a result is simply how many of those outcomes it catches, over how many there are. Both of those numbers are counts of possibilities. And that is the difference worth having on the board. The first formula divides trials by trials. The second divides possibilities by possibilities.

6:03They look almost identical written down, and they are about completely different things. One is a count of what did happen. The other is a count of what could. Now the cost, because the cost is real and it is easy to hide. Take a coin that is secretly bent, so that it comes up heads seven times in ten. Assume it is symmetric and the computation runs perfectly happily and hands you one half.

6:33That one half is worth exactly what the symmetry assumption is worth, which on this coin is nothing at all. Meanwhile the tally is not fooled for a moment. Toss that coin a hundred times and the chance the tally comes out at a half or below is about one in forty-five thousand. Toss it a thousand times and it is under one in a billion billion. So where the assumption fails, the measured value is the one telling the truth, and the computed value is confidently wrong.

7:08Which kills a tempting idea: that the computed number is the real one and the measured number is a rough copy of it. Neither is a copy of the other. They are arrived at in opposite directions. So set them side by side properly. The measured value answers: what happened, in the trials I actually ran? It needs trials, it changes if you run more, and it does not care what you believe about the coin.

7:41The computed value answers: what should happen, if my description of the set-up is right? It needs no trials at all, it never changes, and it is only as sound as that description. Two different questions. Two different things that could go wrong with them. The measured one can be unlucky. The computed one can be misdescribed. There is no rule saying which to trust, and anyone who gives you one is selling something.

8:11What there is instead is a habit: when the two disagree, go and look at the description, because the description is the part that was made up. But they are not strangers, and here is the link. As the trials pile up, the measured value is expected to come close to the computed one. Put a number on that, on a fair coin, and ask how likely a run is to end up a tenth or more away from a half.

8:41Ten tosses: seventy-five point four per cent. Three runs in four miss by that much. A hundred tosses: five point seven per cent. A thousand tosses: under three chances in ten billion. That is what convergence looks like as an actual quantity, and it is why an afternoon of coin tossing gets you a sensible answer. It is also why the die and the coin were the comfortable cases. Trials are cheap, so the measured value gets close.

9:17Now the honesty, and it is a single word. Expected to come close. Not guaranteed to. Narrow the band and watch the promise weaken. How likely is a run to end a hundredth or more away from a half? A hundred tosses: ninety-two per cent. Almost always. A thousand tosses: fifty-four point eight per cent. Still more often than not. Ten thousand tosses: four point seven per cent. About one run in twenty, after ten thousand tosses, is still a whole percentage point out.

9:55And it never reaches zero. No number of trials makes a bad run impossible. There is one more thing that band hides. After ten tosses, the chance of landing exactly on a half is under a quarter, and after a hundred it is eight per cent. So the measured value is almost never the computed value. It is near it, usually, and near is the strongest word available. So: two routes to a probability, and they are not two attempts at one thing.

10:30One is arrived at by doing, and it exists only after the doing. The other is arrived at by assuming, and it exists before anything is done at all. The second is what makes the rest of this subject possible, because most of the questions worth asking are questions you cannot rehearse. But it is only ever as good as the sentence you wrote before you started computing. Get that sentence right and the number is trustworthy.

11:01Get that sentence wrong and the arithmetic will not save you. Every step after it will be flawless. It will just be confidently, precisely wrong, and nothing inside the calculation will ever tell you so.