I used to love R (probably still do), but python became a de facto standard in professional finance. I use a combination of numpy and pandas for my stuff. People who are better developers seem to favor pure numpy-based solutions, but that's more work and I am very lazy.
Thanks for sharing. I have very little experience with pandas. What's the benefit of pandas over numpy? Built in data manipulation functions?
Yeah, a lot of useful stuff is pre-built in. FWIW, we are experimenting with Rust for stuff that has to work fast, but have not fully committed yet.
Interesting. Thanks I will look into pandas a little more. It's mostly for some side projects that require me to do preprocessing on my own.
I like the Box-Cox method of data transformation for normality. Ytransformed = ((Yoriginal^lambda)-1)/lambda see: https://www.wessa.net/rwasp_boxcoxnorm.wasp After the transform and calculation of your result, back calculate the result to the original scale. Yoriginal = (Ytransformed*lambda+1)^(1/lambda)
No, that was deliberate. When someone has to go on CNN and claim there has been a 10 standard deviation event, is that because (a) there really has been a 10 standard deviation event [unlikely, there are too many of those to go around], (b) the distribution is non normal, or (c) because the standard deviation of the distribution is actually much higher today than yesterday? It's a bit of a philosophical question, but is an unusually large return value because the distribution has massive kurtosis, or is it because the vol of the distribution has gone up? Is a negative skewed set of returns because the distribution has massive skew, or is it because the vol of the distribution tends to increase when the market is in an especially bearish mood? Putting philosophy aside; the question is, how well does vol scaling do empirically? (Nice plot by the way). If I do the exercise of rescaling by recent vol across all the markets in my dataset (since an equivalent of VIX isn't available for everything), it nearly always reduce kurtosis and skew significantly. Sure it will never completely get rid of them, but that doesn't mean it isn't worth doing. More importantly, the scaling by recent realised vol produces forecasts for trading strategies that have skew and kurtosis properties that are much nicer than the underlying. To take a simple example, the returns from a risk parity position which is short VIX looks a lot nicer than a flat short VIX position... So I'm not saying you can get rid of this stuff completely, but it does a very good job of flattening things out to produce distributions that are much more pleasant to work with. And it's such a simple step, so why not? GAT PS Haven't used R for about 10 years, mostly python since then... PPS It's ironic that I'm having this discussion at the same time as arguing with other people on the internet who say my trading strategies don't have enough [positive] skew, and I'm not allowed in the pure trend followers club as a result
Here is my review of this book. You need to have quite an advanced background in statistics before reading it. First, you should know all the stuff related to the normal distribution, i.e., mean, standard deviation, Z-score, etc. Also, you need to understand why the normal distribution is not very applicable to trading and finance. Figure out how the distribution of assets' returns differs from normal distribution. Also, learn about long-tail events. After that, you must know nonparametric methods. You would better learn them before reading the book. Hopefully, you'll understand and appreciate their beauty and power. The author tells the readers what to do. He doesn't explain the basics of statistics. From the book, I figured out how to do dynamic position sizing. I am pleased with it.
Would you please explain to me the purpose? As I understand it, non-stationarity in practice means problems with moving averages. The moving averages parameters that were historically optimal will later become very inappropriate and lead to significant losses. You cannot solve this problem with data transformation.
If you really want to get specific, it's the difference between jumps vs ambient volatility. In essence, if you rescale your time-series by the forecast volatility and returns with the sample become perfectly normal, it was a change in volatility regime. Ambient volatility is usually fairly well predicted by the options market, if you need to manipulate forward looking data. For historical data, though, you can just use contemporaneous winsorized volatility - i.e. take the standard deviation of the sample itself after dropping top/bottom extreme points. There are two separate questions, "is it worth doing?" and "does it make my data normally distributed?". The answer to the first one is "yes", the answer to the second one is "not"