I have been wondering... how I should optimize a trading system. I created two trading systems on the euro fx. One works in overnight, and holds positions for 13 hours. The other works in intraday and holds a position for an average of 75 minutes. Once I translated in easylanguage the ideas for the systems, I tested them on Tradestation. The overnight system made money pretty consistently, whereas the intraday system made money on some years and broke even on other years, which was disappointing to me. I then tried to reduce as much as possible the inputs used by both systems and used the "Optimize" feature of Tradestation. This in turn gave me the "best" values for all inputs. However, on the intraday system, the values that made it work best on the test data weren't working well if I used them on two extra years that I had kept outside of the optimization period. Eventually, I figured out (without using "optimize" on the two extra years) settings and values that worked on both periods, within and outside the optimize period. The questions arising now are the following. Since optimize is so misleading and it could cause you to create a system that only works in the past, should I just forget about the "Optimize" feature altogether? Should I just pretend it doesn't exist? What do you think? After trying to use "Optimize" as little as possible, I have actually started using the systems. And they are both losing money, not beyond the expected maximum drawdown but enough to make me wonder whether I abused optimization and created a system that only works in the past. So here I am, losing thousands of dollars, and wondering if I am just being unlucky and happened to start using the system in a drawdown period, or if I misused Optimizing and created a system that only works in the past. An important premise to my questions is that I did manage to create a system that executes, via DDE Excel, exactly the same code that I tested on Tradestation. So, it might be a bad strategy, but what is being executed by Excel and TWS is exactly what was tested on Tradestation (on data from IB). Another important premise is that both systems work on the euro fx (USD/EUR) just as they work on the USD/GBP - which should to some extent exclude over-optimization. Once again, my questions are these. Should you use Tradestation "Optimize" at all? What tricks and methods enable you to avoid over-optimization? Any other advice? Thank you.
Optimization are just that -- curve fitting. Not that curve fitting is not useful; it can be used as a starting point for system design. But ultimately you must analyze and understand why the losers lose, and the winners win.
Hmm, interesting. I should analyze why losers lose... to the point of creating a system that never loses? I don't see how I could ever understand why losers lose. If I did, I would be able to create a system that never loses, which also seems impossible to me. If what you are saying instead is that my system should be based on some coherent ideas, a strategy, a basic assumption I make on the markets such as "they rise on monday", then the answer is yes, of course. I didn't just throw there random inputs and used optimization wildly. As I said I used it as little as possible. For example, I used it in deciding at what time to enter trades (between 15 minutes before the outcry hours and 15 minutes after). It's not like I used 4 moving averages and started optimizing the system based on their crossovers (which always returns a system that makes money).
You may like to read this, I believe Curtis Faith wrote it but am not sure: http://www.tradingblox.com/tradingblox/optimization_paradox.htm
From my experience, the most important thing is to find a strategy that, without any optimization, shows a positive expectancy for as many markets as you can test it on. It doesn't have to be very profitable. Then you can use one or two variables to optimize it.
Thanks to hausse for the article - it makes a lot of sense. I agree with you, telozo - I also thought that I should have as few parameters as possible. Of course, it's not the easiest thing to accomplish. Because, for example I would say that one such system is "buy late on a down day (and viceversa), and sell early on the next day (and viceversa)". And that makes money. But the problem is that you have plenty of room for improvement and that's when you start plugging in more parameters, and then arise my problems and doubts.
IMHO, we can't deal with the optimization problem apart from other system development issues. It's a complex thing, where "black or white" answers rarely can be given. However, there's a very simple way to completely avoid optimization: just put some random rules and parameters into a system. voila, you have a perfectly non-optimized system (but still there's some very very small chance that your "random" rules can be the same as the ones you've come up by optimizing). of course, such system will most likely have a negative performance or flat at best. so it's definitely not just avoiding optimization what we're looking for. In my view a better question is: how to design a system to maximize it's chances of performing in the future, when market characteristics/edges/structure (whatever you call it) will change? If we're talking about single (by single i mean there's no position scaling in/out) systems here, i'm affraid there won't be any simple answer to such question. If we're talking about multiple systems, then the more systems and the less correlation between them, the higher chances of repeating the past performance. This is based on grounds of statistics. Another interesting question: which system has a higher chance of generating let's say sharpe 1.5 in the future? sys "A" - optimized, historical sharpe 2.0 or sys "B" non-optimized, historical sharpe 1.0? and most importantly, WHY? assuming all else being equal, like trade frequency, traded market(s), number and type of rules. A and B only differ in parameter values. I'm curious about your thoughts about this. ok, you haven't use the optimize function here, but still you've "figured it out" manually or by some other method within insample period and then you've checked it on out.of.sample period. How is it different from building a system on one, single, full period? I mean it goes like this: 1. in.sample test OK, out.of.sample bad - sys goes out of the window 2. in.sample test OK, out.of.sample OK - sys is "good". => if you'll test these two systems on one piece of period, you'll come up with the same result - accepting only the 2nd sys. We may feel that the longer out.of.sample performance sys has, the less it's optimized and the higher are it's chances going forward. my problem with this line of reasoning is that the market itself doesn't give a shit whether the system was tested out-of-sample or whether it was traded with real money or not. It's still the same system. It's only psychologically easier for people to accept a system with long "live" performance. back to the essence of the topic: how to avoid the negative effects of over-fitting... i think one has to avoid being tied/dependent on single, specific market's swings or price levels. examples: 1. sys. trades on daily close prices - too much dependent on the single price at the close of the session 2. system holds overnight - too much dependent on overnight gaps, 3. system has very wide stoploss - too much dependent on length of the trends and exit rules, 4. very wide profittarget, also dependent on exit rules, price moves may be too fast for exit rules to capture a profit. 5. MA crossover type of sys without filters - long, tight range period will kill it. etc. that's why i think a trader should take in-depth look into the system's logic/construction rather than it's parameter values or out-of-sample results in order to avoid the over-fitting.
Thanks for your elaborate post, DT-waw. Unfortunately, due to my little knowledge on this subject, I didn't completely understand it. For example, I don't know very much about some of the things you mention such as Sharpe. The words you used got me thinking though, even though I didn't understand some sentences. This part was the clearest to me, so for now I will just comment on this: I think you may have a point. Using out of sample data which is prescribed by many people, even on this forum, may actually do nothing to improve your trading system. I haven't a very good Math background or this would not even be an issue. Unfortunately I am left to wonder about these matters. Also another matter that puzzles me is this - how good is the idea of taking trades only after a large drawdown (say after 2 negative months)? It wouldn't seem to make much sense because you could lose a lot of profit that way, but from time to time I ask myself about it (especially when I encounter those drawdowns and lose thousands of real dollars).
There are two books by Thomas Stridsman that have good chapters on optimization: http://www.amazon.com/Trading-Syste...=sr_1_1?ie=UTF8&s=books&qid=1205013482&sr=1-1 http://www.amazon.com/Trading-Syste...=sr_1_2?ie=UTF8&s=books&qid=1205013570&sr=1-2
If you don't understand simple concepts such as the Sharpe ratio how in the world: a) are you asking questions about much much more complicated notions like optimization? b) do you expect yourself to develop a profitable intraday trading system? c) do you believe that you can compete with other traders who understand those concepts and much more? My questions are intended to serve as constructive criticism. I think you need to do a lot of study before you gamble with money. If you don't know what Sharpe ratio is then you are not trading, you are gambling your money away. Please understand my point and don't get defensive. For most people not making any money in trading by not trading is the best system, in and out of sample. John