i finally finished my system development and started backtesting. I am backtesting using 10 years of ES intraday tick data, and analyzed all the losers (it's an intraday system that only makes a few trades a week, so really isnt that many cases to eyeball over). Anyway, i noticed of all the losers majority took big loss due to my stop loss been set way too high. So i modified the system logic for stop loss to be a lot narrower, for example instead of having a 1:2 reward/risk (take profit once reach 4pt, hit stop loss once reach 8pt). I changed it to 1:1 (4pt profit, 4pt stop loss). This in term increased the p&l of my backtesting significantly. Now the question i have is this curve fitting? I dont want to tweak backtesting so it looks good while failing miserably in production. Thanks
Yes, it is. You went back, say what didn't work and tweaked it until you got the results you wanted. That is like me taking my models back to 1987, seeing that they do poorly on the day of October 19th, and decide to tell them not to trade on that day. Okay, not as bad, but you get the point. Instead, split your historical data up. Three or four fifths for developing your model on, one to two fifths for forward testing your model on. Basically, tape this above your computer: DO NOT DEVELOP YOUR MODEL USING THE SAME DATA YOU WILL TEST IT ON. The best thing you can do is make your risk reward metric adaptive ... but if it has to be static, find what works on the first 3/5ths of your data and see if it works on the last 2/5ths. If it doesn't, then you know you curve fit. Unfortunately, at this point, it will be hard for you to take that step backwards because you already used all your data in development. See if you can get some data from a random time period ... say the 90s ... and see if your system still works.
Newguy, If you only tried changing it from 1:2 to 1:1 then it's statistically very unlikely that you'll be affected significantly by curve fitting. You've only added one degree of freedom to the model, which is insignificant because of how much data you're using: intraday data over 10 years. You're model is still well over-determined by the data. If you tested other risk/reward ratios, say 1:1.1, 1:1.2, 1:1.3 etc then you might have more of a problem. Of course Corey is 100% correct: "DO NOT DEVELOP YOUR MODEL USING THE SAME DATA YOU WILL TEST IT ON." Curve fitting happens during initial development, not just in optimization. I just wanted to clarify that the change you are asking about is not likely to add a large amount of curve fitting beyond what you may already have done. Good luck. Regards, Max
I designed in such a way that it reads es intraday tick data, yet the trade signals are based on price action of at least 3-6 pt movement. That way it makes sure the backtesting is relatively accurate to actual production. i ran the backtesting for 5 years after "curve-fitted" the risk:reward from 2:1 to 1:1. This is based of 3 contracts, it can be scaled up pretty well up to 30 contracts in production as es is very liquid. Next step i am going to hook it up to IB's java api to test on the demo account. Not sure what to make of the result, is it acceptable to have a FULL YEAR with a loss? YEARLY tradeyear , sum(profitloss_dollar) 2002 112.5 2003 10487.5 2004 -4000 2005 5162.5 2006 -1562.5 2007 19225 2008 6625 MONTHLY tradeyear , trademonth, sum(profitloss_dollar) 2002 12 112.5 2003 1 2675 2003 2 1850 2003 3 1800 2003 4 1150 2003 5 -112.5 2003 6 2550 2003 7 512.5 2003 8 -1812.5 2003 9 -1200 2003 10 2075 2003 11 -350 2003 12 1350 2004 1 -575 2004 2 -137.5 2004 3 225 2004 4 -2075 2004 5 -200 2004 6 1537.5 2004 7 -2700 2004 8 450 2004 9 -975 2004 10 2037.5 2004 11 -1262.5 2004 12 -325 2005 1 325 2005 2 287.5 2005 3 887.5 2005 4 2162.5 2005 5 -1137.5 2005 6 637.5 2005 7 -275 2005 8 -862.5 2005 9 1575 2005 10 1500 2005 11 250 2005 12 -187.5 2006 1 -850 2006 2 -212.5 2006 3 1575 2006 4 -2012.5 2006 5 1137.5 2006 6 100 2006 7 -3287.5 2006 8 1375 2006 9 475 2006 10 100 2006 11 50 2006 12 -12.5 2007 1 -500 2007 2 2000 2007 3 2475 2007 4 375 2007 5 987.5 2007 6 1912.5 2007 7 6075 2007 8 5637.5 2007 9 -4687.5 2007 10 3462.5 2007 11 4962.5 2007 12 -3475 2008 1 4475 2008 2 4212.5 2008 3 -1250 2008 4 1050 2008 5 -950 2008 6 -912.5
I don't think you need tick data to do what you're doing. Minute bars should suffice. Adjusting exits is not curve fitting in the destructive sense, it is optmization of you model, which is fine. As long as the model concept is sound then the parameters that dictate profit/loss stops are relatively insignificant. In fact, the best thing you can do is walk forward test your targets/stops and see what happens. Here are the important questions: 1. How many trades (what is your sample size?). For an intraday model over 10 years I'd like to see 500 trades or more. 2. Average profit per trade. Minimum $25 per trade for the ES AFTER COMMISSION AND SLIPPAGE. Use $2.5 for commission and $6.25 slippage per trade. 3. Did you use limit orders? Do not use limit orders unless you have software that can look forward one bar and simulate a fill only when the future price crosses one full tick under or over your limit price. Generally, year end lossess are not acceptable, especially for an intraday model. Without giving away the method, what is the market movement you are trying to capture? Mike
There are folks who generate (and pick from) random systems. Those are the only ones I know of who do not curve fit.
very good question. the one not curve-fitted at all is the system which generates no profit over the long run. the more profit or the more consistent returns it produces - the more it is fitted to the curve. yet 99.9% of traders are unable to grasp this simple concept!
You didn't curve fit your system. You changed the reward:risk parameters from negative (double the amount for loss as there is for the potential gain) to 1 to 1 (same amount for loss as there is for potential gain). Now, provided your system trades with positive expectation, you should do well.