That's what I'm trying to say earlier - curve-fitting/optimization is not a significant factor is systems with large number of trades. jaan: <i>it is perfectly possible to over-optimize a system having any number of historical trades. </i> It's possible, but I think it won't improve system's results. Conclusion is: only systems with small # of trades and large # of parameters are over-optimized just to show good results in testing period. Slippage: It's not easy to calculate possible slippage. First, it depends on market volume/order's size ratio. The higher value this ratio has, the lower slippage will be. If SP Mini 95% of the trading time has >20 contracts on both bid and ask, then when you trade with less than 20 contracts, you can get market orders executions with max slippage of one tick ( on average ). Of course it depends on market data and order execution speed. If you got lagged quotes, or it takes several seconds to submit an order... executions can be poor. Let's say someone wants to trade with 200 contracts. If s/he will submit 10 market orders 20-lot each over the first 3 minutes , AFTER the strategy-signal price is hit, will you be sure, that average execution price of all 200-contracts will always be worse than a strategy-signal price? Many times, after a system indicates: [buy at X price], the price will FALL from that X price-level, thus give you opportunity to buy some contracts at the LOWER prices. Of course, opposite situation also happens, and in these situations you got poorer executions. Let's say system has 50% profitable trades, 50% unprofitable. Then - on average - the price has gone against the direction of the trade in the first minutes after the 50% of trade signals. That gives you opportunity to execute some orders at better prices (assuming that you send multiple orders to open/close the whole position) If system executes orders outside Regular Trading Hours when volume is very low, then slippage calculations are even more complicated. In systems results reports, slippage must be connected with orders sizes, average number of contracts available on bid/ask and tick size. DT-waw
that's not true, really. as an (admittedly contrived) example, if your system has N potential trades to pick, you can apply N binary parameters that tell whether to take n-th trade or not. after some "optimizing" you end up with the ultimate performance: the system picks only winners! adding a parameter will never hurt the (backtesting) performance, it can only make it better. the amount of improvement depends on the "orthogonality" of the parameter (ie whether it adds to the degrees of freedom of the system or not), compared to the existing parameter set. - jaan
jaan: I can't imagine how can you build an optimized system with large # of trades ( >500 ) that has 100% profitable trades! It's easy to optimize system in order to have all 5 trades profitable. It's harder to optimize system in order to have all 20 trades profitable. But it's impossible to optimize a system in order to have all 500 trades profitable! To do this, you must find some pattern that always work, 500 times in tested period! Optimizing is not something bad. The worst thing is when your system is backtested with small # of trades. Results of these systems are worthless. In the next X # of trades the system can produce totally different results. Just like coin flipping. Flip a coin 10 times - your results can be dramatically different in the next 10 flips. It can be: 1) test: 3 tails - 7 heads; 2) test: 7 tails - 3 heads. Now, flip a coin 1000 times. In the next 1000 flips the result will be closer to expected result (500) in % terms. The "worst" scenario can be like: 1) test: 460 tails - 540 heads; 2) test: 540 tails - 460 heads. Gunslinger system has more than 700 trades, so the results tell something about possible system's performance in the future. $50 slippage per trade ( 2 orders open/close ) is to small IMO. But if somebody trades only 1 contract... than it can be unrealistic to set higher slippage. Those are the only things I can tell about gunslinger system. IMO it's better to build your own system(s) rather than buy a single system. DT-waw
for what it's worth: I always assume $100 slip and commission each way per contract. If a SP system cannot handle $200 slip and be net positive, you're going to have a tough time making money with it imho.
DT-wav - Jann's right. # of trades over the backtest timeframe is irrelavent as a gauge of whether a system is over optimized and whether it can produce comparable results going forward. A backtest doesn't really evaluate the effectiveness of a system - it evaluates the effectiveness of the system's ability to be optimized to fit to the dataset over which it's been backtested. There have been tons of systems tried that were optimized during backtesting with impressive results but crashed and burned when they were used going forward. If you want impressive backtest results to help sell a system, just design it to highly adaptive and capable of easily conforming to the backtest data pattern. Then optimize the hell out of it. The final backtest results should look great. Whether that system could ever make any money going forward though you can't tell from those results. Backtest optimization is essentially causing the system to conform to hindsight - which of course is 20/20. You can always add sufficient parameters to effectively "memorize" the backtest data pattern and therefore produce up to 100% accuracy if that's your game. Happens with neural net based systems all the time - when they're overoptimized they basically memorize the intended results of the dataset. The only practical evaluation is a forward test (i.e., a dataset that was NEVER run through the system for optimization) would be to say backtest a system from maybe 1/97-12/99 and then forward test it from 1/00-present. In this case, the results from the forward test would be the only thing meaningful (the backtest results are meaningless due to potential dataset curve fitting or possible implicit memorization of the data pattern). Here's a very simple example of what backtest optimization is like: Define a spectral analysis study. Run it across the backtest dataset and optimize the multiple frequency patterns until you achieve very high correlation of the spectral formula results with the actual dataset. You've basically curve fit the data (you can even do this with effectively random signal noise). But the probability is very high that while you've got high correlation with the past dataset, the ability of the study to predict future price movement (or future random noise as the case may be) will be very poor.
DT-waw, although ArchAngel already answered for me, let me give you another example how to over-optimize a system over large number of trades: let's assume we have a system that averages 10 trades a day over one year of historical data. that's about 2500 trades. now, we just divide the historical data to daily chunks, and apply different set of parameters to each day! this way, from a system of, say, 2 parameters and 10 trades (which is easy to over-optimize, i'm sure you agree) we derive a system with 2500 trades and 500 parameters. which is as easy to over-optimize as each of its 10-trade sub-systems are. QED. of course, in practice no one would do such particular system, but the fact is that the greater the number of (independent) parameters, the greater the system "memory" that can be used to "memorize" historical data. - jaan
jaan & ArchAngel: First, we must define some things. 1) "System optimizing" means that we add parameters to our system in order to get better results. The number of trades of optimized system can be higher or lower than the # of trades of the same system before optimization. 2) Parameters should refer to prices in % terms, not $/point terms. For example: our system has a 20 point stop. Over the backtested period futures price bounce from 800 to 1200 pts. In the next period ( when we trade our system with real money, not backtesting ) prices slide - they bounce from 400 to 600 pts. If we'll have the same 20 point stop, the # of trades will be significantly lower than # of trades in the backtested period. Results will be different, everything will be different. Margins will be lower. More: If we trade with the same # of contracts all the time, the net profit in $ or points can be dramatically different in backtested period and "real trading period" due to different futures prices in both periods. For example: Majority of NQ100 futures systems had higher results in points/$ terms in 1 year period from Nov. 1999 to Nov. 2000 when index price was bouncing from 2,700 to 4,500 over the Nov 2000 - Nov 2001 period when index price was bouncing from 1,200 to 2,700. To compare system's results in two periods, or to compare two different systems results or to compare system's results across different markets - we must use % stats not point stats. As I wrote earlier, slippage ( which can turn system's big profit into zero ) is something that should be set in connection with order size and market volume. Of course, you'll never have all data, all time&sales with sizes for all backtested period. But it should be taken into an account. It's not that easy. 3) <b>All</b> system's parameters must be used in the <b>whole</b> backtested period. How can you <i>"apply different set of parameters to each day"</i>? And parameters can't use time/data from historical charts. "Buy NQ100 futures 1 jan 99 at market, sell 5 Apr 2000". It's not fair. Try to use this system in other market, when [1 jan 99 - 5 apr 2000] was a bear market! 4) System must indicate when to: openLong, exitLong, openShort, exitShort or when to: Buy ( exit short and open long ), Sell ( exit long and open short ). If a system doesn't have rules [when to sell], than we can choose some bull market period and apply only BUY instructions. ---- We have a system with, let's say 2 parameters and 300 trades. In 2 year period it has a net profit ( after slippage, commissions ) 200% of initial capital. We optimize this system by adding more parameters in order to have better % return in the same 2 years period. Let's say after optimization our system has now 20 parameters, 100 trades and 400% return over the same period. ( after adding slippage, commissions ). Our first 2 parameters allow to make 300 trades over 2 years. 20 parameters in the optimized system allow to make only 100 trades over 2 years. So, system's conditions with 20 parameters were met 200 times during tested period in order to make 100 trades ( 100 entries, 100 exits ). Over the next 2-year period these 20 parameters are likely to make similar number of trades, and thus a similar return. However, non-optimized system with 300 trades, in the next 2 year period is likely to make CLOSER (closer in% terms) number of trades to 300 trades, and thus a more closer return to previous return achieved in the first 2-year period. Now, we'll over-optimize this system. Now we have 100 parameters. In the first 2 years they allow to make only 3 trades. In the next 2-years it is quite possible that system's conditions with those great number of parameters won't be met, and a system won't make any trades. The net profit will be zero, which is a huge difference (in % terms) from previous profit from 3 trades. The results may vary not because the system is over-optimized, but because we rely on small # of trades. ---- It is just my theoretical view. ArchAngel - maybe you saw some over-optimized systems that made huge # of trades in backtesting period and than, the same system produce completely different results ( % return ) in the next period. Remember, a "system" can be call a "system" only if it uses conditions and parameters, than can work in any market, at any price levels, over different periods. DT-waw
DT-waw, i'm not sure i want to continue arguing, because you and i seem to be talking about completely different subjects. for example: 1. system optimization in my interpretation is NOT "adding parameters". instead, optimization is finding optimal values for given set of parameters. that's also the common definition, fyi. 2. i'm talking about systems in general, not about some particular sub-class of systems. for example, you require the following: "All system's parameters must be used in the whole backtested period". this means that a system that uses different parameter values for, eg, volatile periods and consolidation periods would not qualify as a valid system in your book. nor would a system that uses different parameter sets for different weekdays. in addition, you require that "Parameters should refer to prices in % terms". this excludes most systems out there. for example, neural networks don't have such clearly defined parameters in the first place, but still qualify as systems in my book. finally, to rephrase, my original point was that you should NEVER judge a system viability based on the number of historical trades alone, unless you know exactly how the system has been trained. i don't seem to find much in your last post that argues this, so perhaps we have reached an agreement? - jaan
Sorry, I've don't understand it correctly. "optimizing" is finding the best values for some set of parameters. Not adding new conditions, parameters. Nothing wrong with that... So, what does the term "over-optimized" mean? Does it mean that: set of parameters have the best values, which produce the greatest net profit? People build trading systems to find the best strategy for some market, to have the higher profit... If optimizing can make a system much more profitable, than it can also make it much less profitable. Which values system-developer should use? Those optimized values, which give high result, or the worst set of values? or maybe something in between? If system results are sensitive to a change in parameters value, than this system is worthless. - i don't remember who said that. but i think it's true. IMO, the more trades system has, the less sensitive it is to change in a parameters values. jaan, of course we should never judge a system only by # of trades, we should look also at net profit, adjusted net profit, max drawdown, max DD/adjusted net profit ratio and more other stats. DT-waw
Hi DT, Maybe I am misunderstanding you, but it sounds like you are assuming that the best parameter settings (i.e. optimized) in the past, as long as there are "a lot" of trades will also be the best parameters in the future, or will somehow remain close to the best. The problem with most commercial systems is that they will run their system on a large data set and optimize for net profit, so that it looks like it survived the test of time, but it is still unlikely to perform the same in the future. You can run a simple test to see this fallacy of this line of thought. Create a simple moving average crossover system on 5 minute bars, so that you get a lot of data. Run an optimization over 8 or 10 months of data to find the best net profit. You will probably get a profitable result with many trades. Now, using the same settings, run it on a few months of data not included in the test set. I would bet that the results will be dismal. The problem with optimization is not in trying to find parameters that may work, but in not fitting them so closely to the data set that they only represent a once in a lifetime occurrance. You don't need 100 parameters to do this. I hate to use optimization. I use it as little as possible, and only to get a general idea of market behavior. When evaluating a system, I want to see profitability accross many parameter settings, on as few parameters as possible. I like to vary inputs, like delaying entry/exit to see if an idea still works. My goal when I create a system is to tear it down. If I can, then I know I would never trade it well. I always assume that future performance will not be as good as past performance, in the event that I accidentally selected curve fit parameters. Jaan and AA have explained it much better than I can. For more info you might consder looking for a book on system design and testing. There are a couple of good ones that explain the pros and cons of optimization. I hope I didn't muddy the water even more. Kirk