I've often asked myself this question about over optimizing my systems and have avoided trying to find optimal parameters, probably to my system's detriment. I am reading a book, The Encyclopedia of Trading Strategies by Jeffrey Katz and Donna McCormick, which provides an excellent treatment of this subject and offers and answer to this nagging question. (Any mathematicians or those better versed in statistics, please feel free to chime in or correct any misstatements that I might make. I am basically talking through something I've just learned...) By performing a t-test on the resulting set of trading Profit and Losses, you can tell what is the likelihood that you have a workable system or just one that you happened upon by chance....or have over-fitted with optimization. I will use two different real world examples to demonstrate this in Metastock: Example 1: I will run two tests over the same data set, a continuous contract of the S&P 500 futures, daily bars, from April 1982 to present. I'll start with the built in Equis: Negative Volume Index w/ Opt system in the System Tester. This system runs a total of 20 optimization steps varying the Moving Average from 10 to 200 and shows a profitable "Points Only" result of 827 points (incl slippage & commissions) over the period versus a buy and hold profit of 747 points. Looks good!!! In the Results.... dialogue, I will right click and copy all of my individual trades for the in-sample test period, in this case 136 trades. These I will paste into Excel, then sort and delete the "Out" trading positions, just keeping the Longs and Shorts. I'll continue my analysis in Excel by finding the mean value of the set of Profit and Loss data generated by my trades, cell formula: =AVERAGE(X1:X137) for example. I'll also calculate the Standard Deviation, e.g. cell formula: =STDEV(X1:X137). Also figure the total number of results that you have, e.g. cell formula =COUNT(X1:X137). Now the t-value is just the MEAN divided by the Standard Deviation divided by the square root of the number of results, e.g. (MEAN/(STDEV/(SQRT(COUNT)))). Higher t-values are better. Using the function =TDIST(t-value,COUNT,1) will give you a probability. This t-test is testing for a statistical difference between two means, in this case, the mean of the P/L and zero, or no profits. Now, I end up with a probability of 10%, so what??? Well, this tells me that there is about a 10% chance that my results or profits were random, or just profitable by chance or luck. The system also has about a nine in ten chance of continuing to work in the out of sample period or in actual trading. But wait, there's more.... Optimization: If I were to take the extra step and consider the optimization on the system, then the results get worse. This optimization consisted of 20 steps, so I'd take (1-.10)=90% chance of success in the future or out of sample data raised to the power of 20, (.90^20)=11% !!! So much for a good chance of profits in the future! This is a great example of overoptimization. Example 2: I'll compare the above system to the Equis: Moving Average Crossovers w/ Opt. Over the same data with the same commission and slippage allowances, the net Points Only profit is 1109 versus Buy and Hold of 747 points. Total trades were 291 for this system. However, this system test used 60 optimization steps to arrive at this improved result...3 times as many as Example 1. The results in this analysis yield a t-statistic of 2.64. The probability of success as calculated above is 99.5%. After adjusting this for the 60 optimization steps, the lowered probability of success or profits is 92%. So even though this system has gone through greater optimization, it looks like it has a greater probability of holding up in the future. Of course, this is just one of several tools you might consider in building that winning system.
WOW !!! Talk about a brain overload Though you have me hooked How long does it take to run a test ? For example the crossover MA's. Can an inter day 1 min chart be tested even for the mini's ? Which sortware testing do you think is suitable for the above, didn't you say you were going to start using Tradestation ? What I've done in the past is usually go back as far as the data allows and try to be realistic on fills etc, then add up winners, losses to see if there is something worth checking out further. Then I try to follow it realtime before testing with $$$, of course I usually ask everyone on Elite first if they know, sorry ;-) Of course when I had RT3 that was only about 10days for 1min charts if I remember correctly, know I should have kept the software a little longer :-( Thanks again, Andy
Andy, The actual backtesting is fairly quick. It is the cleaning up of the data that takes some time. Checking for back ticks and the sort of stuff that can really throw your results off. I will be getting Tradestation in a couple of days. Both Metastock and Tradestation will backtest fairly well, but Tradestation's Easy Language will allow the coding of a more prescise entry condition, like an intrabar entry on a stop. But for testing 1 minute bars for a crossover of Moving Averages, either program will do a very good job. I believe that either program will require you to export results to Excel to do some additional statistical testing, although I know there are add on products like Rina Systems that will do most of this for you. Give me three or four days to pull the data together and I'll test the mini's on a 1 minute basis. It'll be interesting to look through the results.
Thanks Dave, either email me or let me know when your up and running, and I'll give you the info'' I've tried to put together. I'm looking forward to seeing what you think. Thanks again for your time, Andy
I am not sure what you are trying to do, but I would like to know a few things that might clear it up. You said you were using a t-test... what average are you compairing your sample to? Is your sample all of the trades? t=x-u/[q/root(s)] Thanks
Hi Ray, I'm comparing the mean of the profit/loss from the trades with a mean of zero. Essentially, I'm comparing it with a "meaningless result" or rather a profitless system to determine the statistical validity. Yes, all of the trades are included that were simulated by the system for the data series. To have done this more strictly, I would have cut off a certain period, say 1 year of the data set, to use as my out of sample data.
Hi Dave, i have been told that backtesting isnt indictave of future results because the market wont behave in the future as it has in the past. comments anyone? wall
Wallstreeter, That's a tough question that many who are much better than I am at all of this have wrestled with and debated about at length. I agree with you that markets won't necessarily behave the same in the future as they will in the past....however, I subscribe to the notion that they won't change that radically or that rapidly. I think a lot of a markets behaviour is driven by those that trade it, and people's behaviours are very slow to and difficult to change. So while the value of the backtesting data and statistics may be slowly eroding, it provides a good basis from which to trade. This keeps system building and testing a dynamic process. Sure, the turtles breakout systems might not work as well as they did, but I doubt that many would complain about the "transient" performance that it gave. I look for systems that take advantage of a specific patterns or simultaneous conditions which cause people to react. What may be worth it for me to trade may not be worth it to others. I seek a niche. I've seen studies which show that large group behaviour like buying breakouts get more and more ineffective. I do prefer to stay away from those. Different time frames, different markets, and smaller pieces are all ways that I think can keep a system or method under the radar screen of popularity. Also, for my trading, I want probabilities. Even if the market is undergoing change, quantifying my approach helps me pull the trigger then manage a trade. It's just one person's opinion, but I think there's value in taking this kind of approach.
Hi Dave, just checking to see how your doing. Did you ever get Tradestation up and running ? Also how have the futures (mainly the minis) trades been going ? Have you been daytrading them or just posistion trading ? I did get Q-charts up and running and papertested this last week the crossover MA's (NDX-mini). Using worse case fills to try and make it more realistic. Wish I could say I blow it away, but didn't :-( I still ended the week up about 30pt's but with at least 10 trades aday, I'm tired and frustrated, anyway I'll give it the rest of the month to see what it looks like. On paper it'll pay my bills , but in the real world with added slipage and those pesty emotions I'm not sure how well it'll fair On the charts I'm not too impressed with Q-charts, RT3 is a much better system (I think) and I didn't have as many data missing problems as I've already had with Q charts. Nothings worse when your watching 1 min' candles and they disapear for a few min's !!! Who do you use again ? On a question about Globex do you use physical stops ? If so how has the slipage been, I assume they turn to market orders when hit ? Are you still using IB ? I realize it alot of questions so I'll give you a break and stop here, thanks, Andy
Hi Andy, No, I didn't get Tradestation going. I'm still looking to purchase a used copy. My trading right now is centered around very short term positions in the EMinis. I'm currently not doing any position trading. (That's just based on a preference in trading style right now, I've certainly done lots of position trading in the past). I do use QCharts, but mostly just the QFeed that goes to my Metastock program. I have had pretty good luck with the feed but I would recommend that you follow the YahooGroups QCharts maillist. There's a lot of complaining and a fair amount of very specific posts regarding that day's data. But there are also some very good tips and user information that will make life much easier using QCharts. Things like working with the ContinuumClient.dll and some add on programs like SelectServer which are included for free in the files section. It's worth following for information like which servers are using the high speed feeds vs. which are not. There's lots of useful stuff there. For my current trading style, I don't use physical stops. With the IB system, when you do use them, it will send a "synthesized market order" after two prints at your stop price. Since GLOBEX can't accept market orders, the IB system will create limit orders for what I think is 20 points beyond your stop. Because of the liquidity in the ES and NQ markets, the slippage is very minimal, unless it's a rare event, like the recent out-of-cycle drop in interest rates. I'll post some systems stuff in the post that follows. Dave