Squaring removes the sign and punishes the far-out values

0:00The plan has not changed. Take each observation's deviation from the centre, remove the sign, average the results. Exactly one step is different: the sign comes off by squaring instead of by taking a size. And it has to be a square rather than any other even power, because a square is the one that expands cheaply into terms you can total separately - which is what everything after this depends on.

0:30A fourth power would remove the sign just as well and be useless afterwards. So: square the deviations. What can we build out of them? First, notice what squaring buys immediately. A deviation of minus five and a deviation of plus five are different numbers. Squared, they are the same number, twenty-five. Both sides of the centre land on the same non-negative value, and nothing on one side can cancel anything on the other.

1:05That was the whole problem the signs caused. It is now gone, and gone without a modulus. The simplest thing to build is the total. Square every deviation, add them all up, and call that the answer. It has one property already worth having. Every term in that sum is a square, so no term can be negative, so the total can only be nought if every single term is nought.

1:34And a squared deviation is nought only when the observation sits exactly on the mean. So the total is nought exactly when every observation is the mean - when there is no scatter at all. That is not nothing. It is a real property, and it is the reason squares are worth trying. That last statement is an if-and-only-if, and those are easy to believe and hard to test, because on ordinary data neither side of it ever happens.

2:08Take two hundred and sixty records, and deliberately build sixty of them out of a single repeated value. Sixty have a total of nought. Sixty are constant. And the number that belongs to one of those groups but not the other is nought. Had I used only ordinary records, both counts would have been nought and the check would have proved nothing whatsoever. Now the question that decides everything: is that total a measure of spread?

2:41Here is a set of six observations - five, fifteen, twenty-five, thirty-five, forty-five and fifty-five. They add to a hundred and eighty, so the mean is thirty. The deviations are minus twenty-five, minus fifteen, minus five, five, fifteen and twenty-five. Squared: six hundred and twenty-five, two hundred and twenty-five, twenty-five, twenty-five, two hundred and twenty-five, six hundred and twenty-five. Add them and you get one thousand seven hundred and fifty.

3:18Notice that this set reaches twenty-five either side of its centre. Here is a second set: every whole number from fifteen to forty-five. Thirty-one observations, and by symmetry the mean is thirty again. Its deviations run from minus fifteen up through nought to fifteen. And they pair off - for every deviation of a given size on the left there is one of exactly that size on the right. Listed by size, they are every whole number from one to fifteen, twice over, with a single nought in the middle.

3:57So the total of their squares is twice the total of the squares of one to fifteen. Notice: this set reaches only fifteen either side of its centre. Half as far as the first set. Rather than write out thirty-one squares, use the formula for the squares of the first whole numbers: n times n plus one, times twice n plus one, all over six. That formula deserves a moment of suspicion, so I checked it against actually adding the squares, for every count from nought to sixty.

4:36Sixty-one values tried; nought disagreements. At fifteen it gives fifteen times sixteen times thirty-one over six, which is one thousand two hundred and forty. Double it, because the deviations pair off, and the second set's total is two thousand four hundred and eighty. Now put the two side by side, and something has gone wrong. The first set reaches twenty-five either side of its centre; the second reaches fifteen. The first set spans fifty, from five to fifty-five; the second spans thirty, from fifteen to forty-five.

5:18By any reading of the word, the first set is the more spread out. But its total is one thousand seven hundred and fifty, and the second set's is two thousand four hundred and eighty. The total says the second set is the more dispersed. It has the order exactly backwards. The pictures make it plain. Draw both on the same axis, with the same centre marked at thirty. The first set is six dots stretched right across, from five out to fifty-five.

5:52The second is a solid unbroken block of thirty-one dots crowded between fifteen and forty-five. One of these is spread out and one of them is packed together, and there is no argument about which. The total of squares disagrees with the picture. So what went wrong? Nothing subtle. The first set has six observations and the second has thirty-one - twenty-five more. Every one of those twenty-five extra observations contributes another square, and a square is never negative.

6:30The total was measuring spread, and it was also counting observations, and it added the two together. That can be made exact. Append one more observation to any record, and the total goes up by exactly n over n plus one, times the square of that new observation's distance from the old mean. I tried that three thousand three hundred times. The identity failed nought times, and the total fell nought times.

7:02It can only climb. Five of those three thousand three hundred appends changed the total by nothing at all - and all five were the case where the new observation landed exactly on the old mean, adding no distance to anything. But here is the cleanest demonstration that the total is counting. Take any record and write every observation down three times. Nothing about its spread has changed. Same mean, same deviations, same reach from the centre, same everything.

7:39The total is three times as big. Over three hundred records, at three different multiples, that is nine hundred trials, and it failed nought times. A quantity that triples when you change nothing is not measuring what you thought it was measuring. The repair is the obvious one. If the total is partly a count of observations, divide by the count of observations. One thousand seven hundred and fifty over six is eight hundred and seventy-five thirds - two hundred and ninety-one point six seven, rounded.

8:18Keep the fraction in the working, though: two hundred and ninety-one point six seven is a rounding, and truncating instead would print two hundred and ninety-one point six six. Two thousand four hundred and eighty over thirty-one is exactly eighty. Two hundred and ninety-one and change against eighty. The order is back, and it agrees with the picture. That quotient - the average of the squared deviations - is the variance.

8:51It is written with a sigma squared, and read as sigma square. That is the whole definition: take the deviations from the mean, square them, average them. The first set's variance is eight hundred and seventy-five thirds and the second set's is eighty. The first is about three point six five times the second. Before we move on, one honest check. It is no use asking whether the variance ranks two records the way the variance ranks them.

9:25So I scored both measures against something built from neither: how far a record reaches from its own mean. Of three hundred pairs, a hundred and two had the shorter record reaching further than the longer one. The total of squares gets sixty-seven of those hundred and two backwards. The variance gets eighteen of them backwards. Not nought - eighteen. The variance is a far better comparator, and it is not a perfect one, and a video that told you it was would be lying to you.

10:00There is one more thing squaring did, and it was not free. Look again at the first set's total. The two outermost observations contribute six hundred and twenty-five each - one thousand two hundred and fifty of the one thousand seven hundred and fifty. That is seventy-one point four per cent of the whole answer, from two of the six observations. Now measure the same set the older way, by averaging distances rather than squares.

10:33The same two observations carry fifty-five point six per cent. And if you had used fourth powers, they would carry eighty-eight point four per cent. Squaring did not merely remove the sign. It moved a large share of the measure onto the observations furthest from the centre. That is a design decision, not an accident, and it is worth knowing it holds generally. Over three hundred records nobody chose, the outer pair's share under squares came out below its share under distances nought times, and strictly above it three hundred times.

11:14Every single record. But it is not a law about every possible record either. Take nought, twenty, nought, twenty. Its mean is ten and all four deviations have the same size, ten. So the outer pair carries exactly half of the answer no matter which power you use - a half under distances, a half under squares, a half under fourth powers. When every deviation is the same size, squaring has nothing to move the weight towards.

11:49One last thing about what the variance does not tell you. It says nothing at all about where the data sits. Add the same amount to every observation - shift the whole record up the line - and the mean moves by exactly that amount while the variance does not move at all. Three hundred records at three different shifts, nine hundred trials, nought failures. Our two sets are a case of it: both of them sit on thirty.

12:20They are told apart entirely by their spread, which is exactly what a measure of dispersion should do. So: square the deviations, average them, and you have the variance. The squares are still there, though, and their units are the square of whatever you measured - and that is the next thing to fix.