Where this measure breaks down, and why another was needed

0:00We have just built a measure of spread, and worked it three different ways. Fix a centre, take every distance from it, drop the signs, average. It is the most natural measure of spread there is, and it is about to be put down. Three reasons are usually given for putting it down. One: a median can be a poor stand-in for the data, so a measure built about a median inherits that.

0:28Two: measured about the mean instead, the answer is never the smaller of the two. Three: a quantity built out of absolute values cannot be carried into any further algebra. Those three are not of equal weight, and one of them is not really a complaint at all. Taking them apart is the whole of this. Start with the first. A median is the middle of the sorted data, and nothing more is promised.

0:59When the readings sit close together that middle value describes them well. When they are strewn from one end of the scale to the other, it describes nobody. That complaint is easy to state and easy to wave at, so let us measure it instead. Two records, ten readings each, and the same middle value in both. Here are two players' last ten innings. The first, sorted, reads nought, five, thirty, thirty, forty-two, sixty-four, seventy-one, eighty, ninety-one, a hundred and seventeen.

1:36Ten readings, so the median is the average of the fifth and the sixth: fifty-three. He never made fifty-three. His closest innings is eleven runs away from it. Five of his ten ended at forty-two or below, and two of the rest went past eighty, so fifty-three sits in the empty space between two quite different kinds of afternoon. The second player's ten innings have the same median, fifty-three, and this time fifty-three is a score he actually made, three times over.

2:13His closest innings is no distance from it at all. Now push both through the same measure, about that one shared value of fifty-three. The strewn record gives an average distance of thirty-one point six. The steady one gives three point two. Almost ten times apart, from the same centre, with the same count of readings. That is the first complaint, and it is a fair one. Notice what that complaint is actually about, though.

2:45Nothing went wrong with the measure. It reported, correctly, that one record is spread out and the other is not - which is exactly its job. What is unrepresentative is the centre, not the measurement taken about it. So the first complaint is really a warning about medians, and it applies just as much to any other measure you build on one. Keep that in mind; the second complaint has the same shape.

3:16The second reason is a comparison. The same data, measured about the median, and measured about the mean. The claim is that the mean's answer is never the smaller of the two. That is true, and it is worth seeing on real numbers before we ask whether it is a fault. Take eleven readings: three, nine, five, three, twelve, ten, eighteen, four, seven, nineteen and twenty-one. Sorted, they run three, three, four, five, seven, nine, ten, twelve, eighteen, nineteen, twenty-one.

3:53Eleven readings, an odd count, so the median is the sixth one: nine. The distances from nine add up to fifty-eight, and fifty-eight over eleven is five point two seven. Now the mean. The eleven readings total a hundred and eleven, so the mean is ten point zero nine. The distances from that mean add up to six hundred and fifty-two elevenths, and dividing by eleven again gives five point three nine.

4:25Five point three nine against five point two seven. The mean's answer is the larger, by nought point one two. Before we make anything of that, watch the word. The claim is that the mean's answer is never smaller. It is not that the mean's answer is always larger, and the difference between those two matters, because equal cases exist. Take eight readings whose mean is nine and whose median is also nine.

4:57Both answers come out at two point seven five, because they are the same calculation done twice. Take twenty readings whose mean is ten and whose median is ten. Both answers come out at six point two. So the honest statement is: never smaller. And here is where the tidy version of this story is wrong. The tidy version says the two answers agree exactly when the two centres coincide, which is what both of those examples looked like.

5:32So take four hundred records of ten readings each, made by a machine nobody steered, and count. The two answers agree on a hundred and seventy-five of them. The two centres coincide on three. A hundred and seventy-two records agree on the answer while disagreeing about where the middle is. Here is one of them. Sorted, it reads four, twenty-three, twenty-four, thirty, thirty-one, forty-four, forty-six, fifty-three, fifty-four, fifty-six. Its median is thirty-seven and a half, its mean is thirty-six and a half, and both measures come out at fourteen point one.

6:16The true condition is not about the centres at all. To see what the condition really is, stop asking about centres and slide a marker along the line instead. At every position, add up the distances from the marker to all eleven readings, and watch that total. At nought it is a hundred and eleven. It falls to seventy-eight by three, to sixty by seven, to fifty-nine at eight, and to fifty-eight at nine.

6:48Then it turns and climbs: fifty-nine at ten, sixty-two at eleven, sixty-five at twelve. Why does it turn exactly there? Move the marker one step to the right. Every reading now behind you gets one further away; every reading still ahead gets one closer. So the total changes by the step, times the number behind, less the number ahead. It falls while more readings lie ahead than behind, and climbs once more lie behind than ahead.

7:23It turns at the position where the two counts balance - and balancing the counts on either side is the definition of a median. That is not a picture, it is an identity, and it can be checked as one. Over a hundred and twenty records, at every marker position on a whole number or a half, with three different step sizes, the actual change was compared against the step times the difference of the two counts.

7:54Forty-six thousand and eighty steps in all. Forty-five thousand and sixty-five of them landed without a reading inside the step, and on every one of those the two agreed exactly. The other one thousand and fifteen stepped straight over a reading, and the prediction was wrong on all one thousand and fifteen - which is why the rule has to carry its condition with it. Step from eight to eleven on our list, for instance, and you clear both the nine and the ten: the counts at the start predict the total will fall by three, and it rises by three instead.

8:35And the consequence: nothing beats the median. Searching every whole position from nought to forty for a total smaller than the median's, we find none. Searching again in twentieths, so the marker lands two hundred times more often, we still find none. That search is not blind, either - asked the same question about the mean, it comes straight back with three positions that beat it: eight, nine and ten.

9:05So the second complaint is true, and now we know why it is true. Now ask the question the complaint skips. Is being larger than the smallest possible value a defect? The total distance is smallest at the median. That is a theorem, and it means that any measure taken about any other centre must come out at least as large. The mean is one such other centre. So the second complaint amounts to observing that the mean is not the median - which we knew.

9:40And the excess is small. On those eleven readings it is nought point one two, which is two point one nine per cent of the median's answer. Across all eight hundred machine-made records the largest excess I found was twenty-six point three per cent; six records ran past a fifth, and one past a quarter. A measure that is within a few per cent of the smallest value anything could give is not a broken measure.

10:12So the first complaint is about medians and the second is about arithmetic that had to come out that way. Neither of them is why we stop. The third reason is different in kind. It is not about whether the number is good. It is about what you can do with the number afterwards. A measure gets used inside other work: combined across groups, rearranged into a shortcut, differentiated, estimated, carried forward.

10:45All of that is algebra, and algebra needs the expression to open up. An absolute value does not open up. Compare the two ways of killing a sign. Square a deviation and you can expand it. The square of a reading less the centre is the square of the reading, less twice the reading times the centre, plus the square of the centre. Three terms, each of which you can total separately.

11:16Add that up across all the readings and the deviations vanish completely: the total of the squared deviations equals the total of the squares, less the count times the square of the mean. Checked on four hundred records, that failed nought times. Read what that formula does not contain: it has no deviation column in it at all. You never subtract the centre from anything. Now try the same move on an absolute value.

11:48There is no expansion. The size of a reading less the centre is not the size of the reading less the size of the centre, and there is no third expression that fixes it. We can be precise about what makes the difference, and it is a corner. Take a quantity, step a small distance each way from a point, and measure how much it bends there: the value on the right, plus the value on the left, less twice the value in the middle.

12:20For a squared deviation about three, with a step of one, that bend comes out at two - measured at nought, measured at three, measured at seven, the same two every time. That is what a polynomial does; it bends the same amount everywhere. For an absolute value about three, the same measurement gives nought at nought and nought at seven. It is perfectly straight everywhere - except at three, where it gives two.

12:51All the bending is concentrated in one point, and now shrink the step. Divide the bend by the square of the step, which is what a curvature is, and the square gives two, two, two as the step goes one, a half, a quarter. The absolute value gives two, four, eight. It doubles every time the step halves, without limit. That is the corner, measured: not a shape you can see, a quantity that will not settle.

13:22Here is what the corner costs, made concrete. I claimed you cannot recover a combined answer from summaries of the parts. That is a claim about something not existing, and you cannot prove it by failing to find a formula. But you can prove it with two records. The first: nought, nought, two, five. The second: nought, nought, three, four. Four readings each. Both have a mean of one and three quarters.

13:52Both have an average distance from that mean of one and three quarters as well. So any formula built from how many, where the centre is, and how spread they are, sees exactly the same three numbers for both. Now append the same two readings to each - a two and a twenty. Both combined records now have six readings and the same mean. But their average distances come apart: five point one one for the first, five point zero six for the second.

14:25Same input to the formula, two different required outputs. No formula can do that, so no such formula exists. Run the same pair through squares and watch it work. Those two four-reading records have different totals of squares: twenty-nine and twenty-five. That is the summary the squared measure carries, and it is the one that tells them apart. From the two parts' counts, centres and squared spreads alone - never touching the readings - the combined squared spread comes out exactly right.

15:03Over two hundred pairs of records nobody chose, that recipe disagreed with the truth nought times. Write the most natural version of the same recipe for absolute distances, with the square swapped for a size. Over the same two hundred pairs it disagreed two hundred times. Not occasionally. Every single time. So the plan does not change. Take deviations about a centre, kill the sign, average them. Exactly one step is replaced: the sign is killed by squaring instead of by taking a size.

15:44Everything that mattered survives the swap. The signed deviations still cancel to nothing about the mean - checked on four hundred records, nought exceptions - which is why we had to kill the sign in the first place. Every square is non-negative, so the cancellation cannot come back. And squaring is a polynomial operation, so everything the corner blocked is now available. That is the whole reason for the change: not that the old measure gave wrong answers, but that its answers were the end of the road.

16:22The mean of those squares is called the variance, and its square root is the standard deviation, and both are waiting on the other side of that one substitution.