I though I would post results from this simple experiment: pick 15 uniformly distributed random entry times between 9:30 and 3:45 PM. If the entry time is after 9:50 AM, then randomly go "short" or "long" at that time. Exit condition: either exit after N minutes holding time, or when a stop price is reached. I did this on one minute bars on the ES contract for one year using a stop price of 8 ticks (2 SP points) and holding time of 15 minutes. I repeated this random experiment 1000 times on the 1 year of data. Trading only 1 ES contract. Percent profit measured based on the $2500 initial margin requirement at interactive brokers. So $2500 profit would be 100% return. Histogram of net pct profit (1 contract ES minus $4.02 commission per round-trip) attached. Note: in the plot I didn't multiply by 100 so 1.0 = 100 % profit. Mean profitability over 1000 runs: -121.9% annual return Percentage of profitable runs: 11.4% (114 runs / 1000 trials) Mean profit of the top 11.4% runs: 52.7% annual return Best profit: 194.6% annual return So predictably, 88.6% of these "random traders" lost money in the last year while 11.4% made money, and the best money maker made a pretty good return close to 200%.
Histogram of win rate Here mean win rate was not 50% -- it was 47.4%. I believe this is systematic and is due to the stop condition. If I remove the stop condition, it will be closer to 50% (i.e., just hold 15 minutes and exit regardless of price). Even with this skew, the "best" systems exhibited a 53% win rate by chance.
I could repeat this experiment with no stop (not sure when I will get to it). But the lesson is clear. If one automatically sought out systems with high profit factors or total profits or sharpe ratios etc, just searching only 1000 random systems gave me pretty good results. Imagine what would happen if I did this millions of times. Will I find an even better system in the tail of the distribution? I would say it's pretty much guaranteed. And it will also be guaranteed to be useless.
This is a good illustration of data snooping bias. There are software algorithms such White's Reality Check that calculate the bias within a set of algorithms. Algos must then exceed this bias for having significance. http://4xtutor.com/autotrade/maths/data-snooping-and-whites-reality-check-part-1-introduction/
Actually there was one small error in my sim: I wasn't sorting the random entry times correctly so it was on average only trading 2-3 trades per day. So I will just reword the system as entering 2 to 3 times per day at random (rather than around 15), but same exit conditions. Basically the conclusions should remain similar because increasing to around 15 trades per day will simply increase the sample size from around 700 trades over the year to more than 3700 trades.
make a target profit of 100$ and disaster stop loss of 500$, skew in win rate will reverse and number of profitable results should go up considerably. Cheers