That's an interesting perspective. I was originally trying to show that a system that should NOT work (totally random entries) can be made to appear to work if one searches (aka optimizes) long enough. In fact around 10% of those systems made money despite being totally random entries. But perhaps you are coming from this angle: if we kept the entries random, and optimized the exits, will I be able to skew that distribution in my favor? Well, another approach is to ask: if I randomized both entries and exits, then will I find systems that make a lot of money? The answer is yes. But in either case, these are random systems, so they are not repeatable. The performance of these are pure curve fits.
Yes you got it. That was my line of thinking. I am positive it will lead to good in sample results for a majority. Might even lead to few good results out of sample but needs more thinking. Cheers
Do I miss something here? What does the reality check have to do with the random system tested? The system wasn't selected as a best performer from a data-mining process. Can you explain what you mean here?
When you test a strategy, you usually don't know if positive results are just luck, or if you really have an edge. This experiment shows that just luck can generate very positive results. The Reality Check finds such issues by giving a profit threshold that a system must exceed when it really has an edge, i.e. is not random.
Yes that was the goal. It is the null hypothesis to compare against. I could also run the random exit scenario as well. But this alone may allow you to assign a p-value on your sharpe, win rate, r, pf, etc. (e.g., if you get a pf of 1.2, you should be very aware that it is very possible to get that by chance. If you have a sharpe ratio of 2.0, again very possible that you got lucky).
Trading systems are proven in actual trading, not by any reality check someone developed and hypes. These tests have very high type II error. "Specifically, WRC permits the data miner to develop the sampling distribution for the best of N-rules, where N is the number of rules tested, under the assumption that all of the rules have expected returns of zero. In other words, WRC generates the sampling distribution to test the null hypothesis that all the rules examined during data mining have expected returns of zero.â What does this have to do with trading system development? I do not care what one million rules do if I can find a rule that works. If you keep reading this staff and believe in it you will never trade. The real intention is to scare traders away. Fear makes money for some people. It is like religion. "If you do this, you will go to hell". The author of the above excerpt sounds like a priest to me. I wonder if he has even traded 100 shares in his life.
Of course, the flip side is this: if you are optimizing a new method that you think has predictive value of good entries versus bad, and if you are using the same exit condition as I outlined above, and if any of your performance measures far exceed the 95 percentile of the shown distributions, then perhaps there is a good chance that it is a significant system (i.e., it is not due to luck). If you have different exit conditions, then one must simulate exactly those conditions with randomized entries. By doing so, one has some level of confidence that the system performance is not a fluke.
The better the system performs out-of-sample, and the bigger the out-of-sample test itself is (# trades out-of-sample versus # trades in-sample), the more comfortable you can become that the results are not randon, and that the edge is real. No?
I am not so sure about this. Let's say you ran 1000 OOS tests while optimizing a parameter in your system. Out of those 1000, I think you can still expect around 100 of them to make money due to shear chance. Again, if your performance metric is significant beyond p<0.05, then you have slightly more confidence, I think.