After not being successful at discretionary trading, I've decided that I will not trade any more until I have a system that has backtested positive. I just watched a video on you tube about the trade ideas oddsmaker. They make it look so simple. In just 8 minutes they produced a strategy that was 60% successful with a near 1:1 reward/risk ratio. This should, in theory, be positive. But what about in real trading? I'm a bit skeptical because it just seems too easy. Let's say I fork out the $1100 for oddsmaker and tweak the strategy to give a slightly better reward risk ratio, can I expect it to be positive with real trading? I know that getting filled is the biggest variable. What are some other reasons that one might not receive the same performance as the backtests indicate? I'm not just talking about trade-ideas, I'm also doing tests with NinjaTrader. That way I can test with more than 2 weeks of data, plus it's free. I'm looking forward to hearing others' experiences with automated trading, especially whether you are getting the same performance as the backtesting suggests.
They better match. If not, it's time to turn off the strategy, go back and figure out why, voila! another lesson learned on the road to consistency.
The problem with backtesting is that the results are optimized to make the trading system look good. Optimized trading systems rarely work for very long in real time trading.
If you're doing a good job not biasing your backtest and you're doing a sufficient trades to have confidence in your metrics, you can get about 90% of what your backtests predict. That said anyone selling you trading advice knows they can't actually trade off it (or atleast don't have the confidence to trade off it - hint: you shouldn't have any confidence in it either.) I'm not sure what the absolute limits are but to trade a system, it needs to produce maybe 100 (1000 would be better) trades in a period it wasn't trained against and the average results of those trades has to at least approximate how well it did on the period it was trained against or you've got nothing. (btw, if you've got something here you don't sell it and you don't even tell anyone you have it.) Long story short: trading is hard, system trading is harder and anyone selling you a system is a fraud.
Thanks communistmonkey, very interesting comments there. Especially about testing the system with data that it wasn't written against. I watched this video from trade-ideas: http://www.youtube.com/watch?v=FhzftR6B4ig They show a system that is 41% winning trades but has around 3:1 reward risk ratio which gives it positive expectancy. The only downside I see to this oddsmaker thing is that it only backtests with 2 weeks of historical data which doesn't seem that much to me. For them it was 300 trades. But then one could either wait 2 weeks and retest it to get a new 2 weeks of data, or one could forward test it using a sim account. So my thinking is, why isn't everyone doing this? The skeptic part of me says "this is too good to be true" but the engineer part of me says "the numbers don't lie". I'm thinking this is going to cost me $1100 to find out. Which is why I ask here first..
No, this is what the software industry wants you to think. This is a quote from Michael Harris' new book "Profitability and Systematic Trading": "one severe theoretical limitation [of backtesting] arises from the fact that when a trading system is back tested it is not a real market participant, and thus its effect on market prices is not reflected in the results." Optimization is a secondary limitation, not a primary one. If your objective function stays near a maximum or minimum for a wide range of parameter values then optimization may be good, not evil. At last, someone is honest enough to say that. Harris also talks about several other limitations of backtesting having to do with software limitations and data adjustments. I highly recommend that book: http://www.wiley.com/WileyCDA/WileyTitle/productCd-047022908X.html Ron
My 2 cents. I use the OM. Have been an early adopter. It is 3 weeks worth of data fyi. In each day you can have a maximum of 100 trades per day. So on any 15 day cycle you can have about 1500 trades to sample from. That is a decent sample size. But once you start adding filters to your system like volume in 5 minutes etc the sample size gets smaller. There is no magic in what they do. It is just number crunching. I think the benefit comes from not having to write a possible 1500 scripts to test a scenario on 1500 possible different stocks. It does not mean the trend is necessarily going to continue exactly like it had, but I think it helps you make the better educated guess. That is all we do is traders
Systems in trading are like their brothers and sisters in the casino. When a customer tells a casino host, he/she is s systems player, the casino's limo driver is waiting for them at the airport.
For what it's worth 2-3 weeks of data seems too small to be useful. 2-3 months seems like the minimal amount. If that's their freebie amount then it's understandable, but if you're paying $1100 for access to only 3 weeks of data: a) they aren't giving you enough data to do anything with b) they probably aren't giving you enough data because it will expose their systems as useless