diligent, You may want to check this out: http://en.wikipedia.org/wiki/Kurtosis In particular, read about leptokurtosis or leptokurtic distributions.
perfect - thank you. "A distribution with positive kurtosis is called leptokurtic. In terms of shape, a leptokurtic distribution has a more acute "peak" around the mean (that is, a higher probability than a normally distributed variable of values near the mean) and "fat tails" (that is, a higher probability than a normally distributed variable of extreme values)"
[in case it helps anyone on this: ] If you want to understand how infinite variance is possible, you first have to make sure you are clear about the meaning of variance. Variance can be expressed roughly as "the mean of the squares of the deviations of each data point from the mean of all the data points". So in order to define variance properly, you first have to define "mean". People tend to just equate mean with average, but as we all learnt at school, there are three different types of average and mean is only one of them. For a discrete set of numbers, it's just the sum of the numbers divided by how many there are. When we look at a theoretical distribution, e.g. normal, Cauchy, Poisson, etc., we're not looking at a discrete set of numbers, but a continuous distribution. Here we have to take an integral rather than a sum. The integral which defines the mean, in general, is I(x.p(x) dx) where p is the probability distribution and I replaces an integral sign. Since a variance is a mean of something (the mean of the squares of the deviations), integrals for variance have the same form. Now integrals are effectively infinite sums, and they don't always yield finite numbers; they can "diverge". The most important "divergent" infinite sum is the harmonic series: 1/1 + 1/2 + 1/3 + 1/4 + .... . This sum never reaches a limit; the more reciprocals you add, the slower it goes up of course, but it never stops. Analogously, the integral of 1/x between x=1 and x=infinity is undefined; it is infinite. Well, in just the same way, some of the theoretical distributions mentioned, such as the Cauchy distribution, have variances which are undefined. I'll let someone else explain the philosophical signicance of infinite variance ; but I think it's also worth mentioning that it's a big assumption that, say, prices follow any specific distribution at all (that's the assumption of a "stationary stochastic process" or some such). Btw I appreciate the OP's post about Chebyshev, interesting to think about .. I tend to doubt its value in application to financial time series for the reasons already mentioned.
Mentioning the Cauchy distribution can actually be of some help. I was able to get some more information. Thanks. Yes, I know that an integral can diverge, but it doesn't have to. Not having some calculations to look at, I was looking at what the meaning of variance should be, and also the meaning of mean, for that matter. I like the definition of Wikipedia: "In probability theory and statistics, the variance of a random variable (or equivently a probability distribution) is a measure of its statistical dispersion, indicating how its possible values are spread around the expected value. Where the expected value shows the location of the distribution, the variance indicates the scale of the values." Now, just intuitively seen, an infinite variance sounds just weird, as well as an infinite mean. The fact that the strong law of large numbers doesn't apply (with an infinite mean that makes kind of sense) looks also strange. And I still wonder if there is a "philosophical significance", as you said. Absolutely. You probably have to consider a process dependent on more variables. Thanks for the post
Yes, I understand it's still not easy to imagine infinite variance. The problem is that the market is not solid like a metal rod in a science experiment (where we use variance just as a way of assessing our measurement inaccuracy), but it also seems that the market is not like the distribution of human heights (where millions of independent additive effects appear to allow the incredibly powerful Central Limit Theorem to apply, so that we get a nicely tractable Gaussian); and so these philosophical questions about the difference between a constructive measure of variance as "the mean of the square of the deviations from the mean of the data" and the true variability of the object become paramount. It might be better to think of it as infinite variance is "undefined variance"; i.e. the concept of variance doesn't apply in this case. There are other measures of course - maybe it's better to use mean absolute deviation rather than standard deviation (sqrt variance) in some cases. Not sure. I remember always being puzzled about why st. dev. was preferred to that when I learnt basic stats. I think it comes back to the obsession with Gaussians again.
From wikipedia: "The average absolute deviation from the mean is less than or equal to the standard deviation. One way of proving that relies on Jensen's inequality." Their example cites the series: 2,2,3,4,14. The mean absolute deviation ends up being 3.6 using mean as measure of central tendency, whereas the standard deviation is 5.09902. Something I also found interesting was also that the measure of central tendency affects measurements of deviation; I typically use the mean. Also from wikipedia: "The average absolute deviation from the median is less than or equal to the average absolute deviation from the mean. In fact, the average absolute deviation from the median is always less than or equal to the average absolute deviation from any other fixed number." As I understand it, I'd want a larger (pessimistic) value for a deviation measure, hence using STDEV and AVG. For me these are somewhat 'soft' numbers anyway, and putting these ideas together in this way allows me to apply it.
Thanks for the references and details. Interesting stuff. But I'm not sure I buy the argument as to stdev being more pessimistic and therefore better. If that were the reason to use sdev (sqrt(variance)), then why not use the 24th root of the mean of the 24th powers of the deviations? This is guaranteed to be larger than or equal to the standard deviation. Do you see what I mean? Honestly I think it's to do with the Gaussian/normal distn. obsession.
Actually, people do. In the field of nonlinear optimization, the "Least Pth approximation" applies exactly this idea (raising deviations to the power P, summing, and taking the Pth root). Note that in the limit as P approaches infinity, this calculation merely returns the largest of the individual deviations. In other words, it is a continuous approximation of the (discontinuous) "MAX" function. Since the fastest optimization algorithms require the objective function to be continuous (with continuous 1st and 2nd derivatives too), the Least Pth Approximation is incredibly valuable; it lets you use the best optimizers on minimax problems. Do a google search for Least Pth Algorithm and/or Least Pth Approximation. It's good stuff. By the way, there was a Russian mathematician who studied minimax optimization quite extensively, and derived a set of equiripple polynomials to study it. These polynomials are in active use, today, in your stereo CD player's D-to-A converter and filter (among thousands of other places). The mathematician's name? Chebyshev. Yes, the very same guy.
"Fat tails" refer to a distribution... Where common events like a 1% move follow the Normal Distribution... But extreme events like a 10% move are more common than the Normal Distribution would indicate. So a good fit for financial markets... Is a ** simple ** custom distribution... Comprised of the Normal Distribution for +/- about 2 standard deviations... And custom values beyond +/- 2 standard deviations... To account for higher probability of extreme events... or "fat tails". All this has been widely known for at least 25-30 years... And all the pros are running powerful computers with well-designed custom distributions... So the options market has been efficient for a long time. Taleb has not discovered/revealed anything new... Just wrote a great, accessible book that everyone should read and understand. Tweaking distributions is so marginal... That it is dwarfed by an order of magnitude by execution... (How professional your trading and technology is... and how low your costs are). Taleb has probably been buying volatility from Niederhoffer the last few years... So his results likely are the inverse of Niederhoffer's great, but "lucky coin-flipper", results.