Part of an article in The Economist of 24 Jan 2009 entitled “In Plato’s Cave”, deals with Value at Risk - what it is, what it is used for and how it is arrived at. I am dumbfounded and appalled; dumbfounded that people in positions of great importance should be so ignorant and appalled that such ignorant people could ever have attained such exalted positions. The chief finance officer of Goldman Sachs is presumably paid sundry millions of dollars; clearly he is not up to the job of secretary of the local golf club.
To make it clear that I have not misrepresented the situation I shall copy the entire relevant section of the article.
“Almost as damaging is the hash that banks have made of “Value at Risk” (VAR) calculations, a measure of the potential losses of a portfolio. This is supposed to show whether banks and other financial outfits are being safely run. Regulators use VAR calculations to work out how much capital banks need to put aside for a rainy day. But the calculations are flawed.
The mistake was to turn a blind eye to what is known as “tail risk”(i). Think of the banks’ range of possible daily losses and gains as a distribution. Most of the time you gain little or lose little. Occasionally you gain or lose a lot. Very rarely you win or lose a fortune (ii). If you plot theses daily movements on a graph you get the familiar bell-shaped curve of the normal distribution (see chart 4) (iii). Typically a VAR calculation cuts the line at say, 98%or 99%, and takes that as the measure of extreme losses.
Tail Spin
However, although the normal distribution closely matches the real world in the middle of the curve, where most of the gains and losses lie, it does not work well at the extreme edges, or “tails”. In markets extreme events are surprisingly common- their tails are “fat” (iv). Benoit Mandelbroit calculated that if the Dow Jones Industrial Average followed a normal distribution, it should have moved by more than 3.4% on 58 days between 1916 and 2003, in fact it did so 10001 times. It should have moved by more than 4.5% on six days; it did so on366. It should have moved by more than 7% only once in every 300,000 years; in the 20th Century it did so 48 times.
In Mandelbroit’s terms the market should have been “mildly” unstable. Instead it was “wildly” unstable. Financial markets are plagued not by “black swans”- seemingly inconceivable events that come up very occasionally- but by vicious snow-white swans that come along a lot more often than expected (v).
That puts VAR in a quandary. On the one hand you cannot observe the tails of the VAR curve by studying extreme events because those extreme events are rare by definition. On the other you cannot deduce very much about the frequency of rare events from the shape of the curve in the middle. Mathematically the two are almost decoupled (vi).
The drawback of failing to measure the tail beyond 99% is that you leave out some reasonably common but devastating losses. VAR in other words is good at predicting (vii) small day to day losses in the heart of the distribution but hopeless at predicting severe losses that are much rarer- arguably those that should worry you most.
When David Vanier, chief financial officer of Goldman Sachs told the Financial times in 2007 that the bank had seen “25 standard deviation moves several days in a row”, he was saying that the markets were at the extreme tail of their distribution. The models did not begin to predict that the tails would move so violently. He meant to show how unstable the markers were (viii). But he also showed how wrong the models were.”
This whole rigmarole is not simply wrong; mathematical statistics is simply inapplicable. An insurance actuary does not work out the risk of your house burning down from some mathematical distribution but from how frequently houses have burnt down in the past. The daily market movements of the Dow Jones, the Nikkei, the FT are known – all of them. We know about the whole “population”. If the Dow Jones moved by more than 3.4% on 1001 days between 1916 and 2003 that is a fact. If there 17000 daily movements of the Dow in that time, 5.8% of them were by more than 3.4%- no standard deviation, no probability, no distribution, no mathematics, no doubt. Of course the future will be different from the past but that is not relevant to the estimation of VAR.
I should stop there; nothing more needs to be said; the exercise should never have been attempted. However, given that an attempt was made, if those who carried it out had known even a little statistics they would not have reached such ridiculous conclusions. But then if they had known even a little statistics they would not have started.
Let me make quite clear what statistics is about. It is about saying something about everything – the population – from a sample. But the sample is always the best guide. Mathematical statistics can only tell you how reliable the estimate from the sample actually is. The mean of a sample of say heights of male Englishmen is the best guide to the mean height of all Englishmen. Statistics only tells you what chance there is of your figure from the sample being wrong and by how much. For example, suppose a sample gives a mean of 5ft 8 in; statistics might tell you that there is a 99% chance of the mean height of all Englishmen being between 5ft 7 in and 5ft 9 in. Such estimates are based on certain mathematical distributions that are defined by equations. Those distributions are idealisations, they never occur exactly in the real world.
The attempt to deduce the probability of a large daily fall in the stock-market from a theoretical distribution is, as I hope pointed out, quite pointless since the best guide is the figures of daily falls. Whether of all falls or only a sample. However, the V A R would not have been so misleading if those who engaged on this bizarre exercise had understood what they were about.
Since they assumed that the daily changes followed a normal distribution I need to say something about the normal. The normal distribution will occur with the frequency of heads and tails in a large set of long series of unbiased tossings of an unbiased coin. But there is no totally unbiased method of tossing and no totally unbiased coin. As the set becomes larger and the series longer the distribution becomes an equation of the normal; the equation is in fact a negative exponential. The normal is defined by 2 things – by the mean and by the square of the difference of all the values from the mean, called the variance. Given the mean and the variance there is only 1 normal. If you divide the variance by the number of values and take the square root you get the Standard Deviation (S D). Only 5% of the normal values lie more than 2 S D from the mean and only 1% more than 3 S D.
You can calculate the mean and the variance of any set of numbers – say the last three numbers in the telephone directory- but that says nothing about whether they are normally distributed.
If we were to use the normal as is done with VAR, to estimate the chance of stock-market falls greater than a certain amount, we would need to do 2 things; firstly determine that the daily falls were distributed normally and secondly judge how accurate is our estimate of the mean and the variance of the daily falls. To go back to our ideal coin this has 2 characteristics: – heads and tails are equally likely and one tossing is independent of all other tossing’s- however long the series of heads the chance of a head remains a half. Neither is the case with stock market changes. The large falls occur more often than large rises – stocks climb by the stairs and go down by the elevator - and one bad day is often followed by another. So we would expect stock market changes not to follow the normal. And they do not. The actual distribution as Mandelbrot found, is nothing like normal. By the way it does not take a world-renowned mathematician to show that. Any schoolboy with basic arithmetic could do the job. But even if the distribution of rise and falls had been not obviously different from normal there is no way of gauging the reliability of our estimate of the S D. As the sample size increases the estimate of the S D must become better but there is no way of knowing by how much; the S D does not get smaller as the sample size increases. (The mean is different as my note on Basic Statistics explains.)
If a statistician did use the normal in this context it would be to argue that the distribution was not normal. If one found a fall of say 4 S D the statistician might say “There is only one chance in chance in 10000 that the observation came from a normal distribution – with mean A and S D B.” In simple terms the daily changes are not normally distributed. But then who would ever have thought that they were – except an economist.
But let me repeat, there is no point in trying to fit daily changes to any distribution. You can never do better than the raw figures; statistics can’t turn a sow’s ear into a silk purse.
Here are a few comments on the text but these are details, although they do serve to reinforce the ignorance of everyone in the business.
(i) It is not “tail risk” whatever that may be, but that the “tails” never fit anyway even with things like heights of men (or women) that are close to normally distributed.
(ii) I suspect that quite often you lose a fortune but never make a fortune. I believe the saying goes, “Shares climb by the stairs but go down by the elevator”.
(iii) That is simply untrue. The daily changes are as Mandelbrot showed. It is not even true
(iv) that small changes fit the normal. The full distribution of actual changes must lie below the normal for small changes since the actual changes lie above the normal for large changes; the area under the 2 curves must be the same since that area must come to 100%. In technical terms, compared with the normal the actual distribution is platikurtic and negatively skewed- flatter and not symmetrical. Both can be tested for.
(v) All this shows is that the normal does not fit. In fact since you have all the daily readings, the markets have been plagued by what they have been plagued by- end of story, as I pointed out at the start.
(vi) I do not know what mathematically decoupled means. The author still does not seem to understand that the normal simply does not fit at all.
(vii) This applies again here. A minor-but perhaps not so minor-point is that the distributions do not “predict” at all; they merely represent or purport to represent, the past.
(viii) I assume the Chief Finance Officer of Goldman Sachs was claiming he could hardly be blamed for failing to anticipate an event much less likely than the earth being struck by an asteroid. But if an umpteen billion to one chance happens day after day as well as at other times in the last century, one might wonder if the event is really so unlikely. In fact statistics does not argue that way; what statistics says is that there is only 1 chance in a billion of the event coming from a normally distributed population with a mean of x and an S D of y. To go back to the beginning, the best guide to the frequency of catastrophic events is how often they have happened.

I am surprised that banks need to guard only against isolated daily falls, not against sequences of bad days – not against rainy days but against wet months and wet years. Even if there were only rainy days-that is if daily changes were independent of each other-there would still be a chance of a run of bad days – the chance of a run of sixes when throwing a die. As it is there are recessions and depressions. I would have supposed it was those for which banks needed to make provision. Statistics can say little about when the next downturn will occur and how deep it will be, nor for how long the current recession will last.
If the calculation of VAR is such a misleading farce one wonders about other “mathematical” models that banks and investors have relied on and whether it is these that in large measure have caused the current financial catastrophe.

P SYMMONS Feb 2009
PS I have attempted to elaborate on the statistical argument in another piece- “Basic Statistics”.