did i apply curve fitting to my system

Discussion in 'Automated Trading' started by newguy05, May 3, 2009.

  1. i finally finished my system development and started backtesting.

    I am backtesting using 10 years of ES intraday tick data, and analyzed all the losers (it's an intraday system that only makes a few trades a week, so really isnt that many cases to eyeball over). Anyway, i noticed of all the losers majority took big loss due to my stop loss been set way too high.

    So i modified the system logic for stop loss to be a lot narrower, for example instead of having a 1:2 reward/risk (take profit once reach 4pt, hit stop loss once reach 8pt). I changed it to 1:1 (4pt profit, 4pt stop loss). This in term increased the p&l of my backtesting significantly.

    Now the question i have is this curve fitting? I dont want to tweak backtesting so it looks good while failing miserably in production. Thanks
     
  2. Corey

    Corey

    Yes, it is. You went back, say what didn't work and tweaked it until you got the results you wanted.

    That is like me taking my models back to 1987, seeing that they do poorly on the day of October 19th, and decide to tell them not to trade on that day.

    Okay, not as bad, but you get the point.

    Instead, split your historical data up. Three or four fifths for developing your model on, one to two fifths for forward testing your model on.

    Basically, tape this above your computer:
    DO NOT DEVELOP YOUR MODEL USING THE SAME DATA YOU WILL TEST IT ON.

    The best thing you can do is make your risk reward metric adaptive ... but if it has to be static, find what works on the first 3/5ths of your data and see if it works on the last 2/5ths. If it doesn't, then you know you curve fit.

    Unfortunately, at this point, it will be hard for you to take that step backwards because you already used all your data in development. See if you can get some data from a random time period ... say the 90s ... and see if your system still works.
     
  3. maxdama

    maxdama

    Newguy,

    If you only tried changing it from 1:2 to 1:1 then it's statistically very unlikely that you'll be affected significantly by curve fitting. You've only added one degree of freedom to the model, which is insignificant because of how much data you're using: intraday data over 10 years. You're model is still well over-determined by the data.

    If you tested other risk/reward ratios, say 1:1.1, 1:1.2, 1:1.3 etc then you might have more of a problem.

    Of course Corey is 100% correct:
    "DO NOT DEVELOP YOUR MODEL USING THE SAME DATA YOU WILL TEST IT ON."

    Curve fitting happens during initial development, not just in optimization. I just wanted to clarify that the change you are asking about is not likely to add a large amount of curve fitting beyond what you may already have done.

    Good luck.

    Regards,
    Max
     
  4. What system is not curve-fitted?
     
  5. I designed in such a way that it reads es intraday tick data, yet the trade signals are based on price action of at least 3-6 pt movement. That way it makes sure the backtesting is relatively accurate to actual production.

    i ran the backtesting for 5 years after "curve-fitted" the risk:reward from 2:1 to 1:1. This is based of 3 contracts, it can be scaled up pretty well up to 30 contracts in production as es is very liquid.

    Next step i am going to hook it up to IB's java api to test on the demo account.

    Not sure what to make of the result, is it acceptable to have a FULL YEAR with a loss?

    YEARLY
    tradeyear , sum(profitloss_dollar)
    2002 112.5
    2003 10487.5
    2004 -4000
    2005 5162.5
    2006 -1562.5
    2007 19225
    2008 6625


    MONTHLY
    tradeyear , trademonth, sum(profitloss_dollar)
    2002 12 112.5
    2003 1 2675
    2003 2 1850
    2003 3 1800
    2003 4 1150
    2003 5 -112.5
    2003 6 2550
    2003 7 512.5
    2003 8 -1812.5
    2003 9 -1200
    2003 10 2075
    2003 11 -350
    2003 12 1350
    2004 1 -575
    2004 2 -137.5
    2004 3 225
    2004 4 -2075
    2004 5 -200
    2004 6 1537.5
    2004 7 -2700
    2004 8 450
    2004 9 -975
    2004 10 2037.5
    2004 11 -1262.5
    2004 12 -325
    2005 1 325
    2005 2 287.5
    2005 3 887.5
    2005 4 2162.5
    2005 5 -1137.5
    2005 6 637.5
    2005 7 -275
    2005 8 -862.5
    2005 9 1575
    2005 10 1500
    2005 11 250
    2005 12 -187.5
    2006 1 -850
    2006 2 -212.5
    2006 3 1575
    2006 4 -2012.5
    2006 5 1137.5
    2006 6 100
    2006 7 -3287.5
    2006 8 1375
    2006 9 475
    2006 10 100
    2006 11 50
    2006 12 -12.5
    2007 1 -500
    2007 2 2000
    2007 3 2475
    2007 4 375
    2007 5 987.5
    2007 6 1912.5
    2007 7 6075
    2007 8 5637.5
    2007 9 -4687.5
    2007 10 3462.5
    2007 11 4962.5
    2007 12 -3475
    2008 1 4475
    2008 2 4212.5
    2008 3 -1250
    2008 4 1050
    2008 5 -950
    2008 6 -912.5
     


  6. I don't think you need tick data to do what you're doing. Minute bars should suffice.

    Adjusting exits is not curve fitting in the destructive sense, it is optmization of you model, which is fine. As long as the model concept is sound then the parameters that dictate profit/loss stops are relatively insignificant. In fact, the best thing you can do is walk forward test your targets/stops and see what happens.

    Here are the important questions:

    1. How many trades (what is your sample size?). For an intraday model over 10 years I'd like to see 500 trades or more.

    2. Average profit per trade. Minimum $25 per trade for the ES AFTER COMMISSION AND SLIPPAGE. Use $2.5 for commission and $6.25 slippage per trade.

    3. Did you use limit orders? Do not use limit orders unless you have software that can look forward one bar and simulate a fill only when the future price crosses one full tick under or over your limit price.

    Generally, year end lossess are not acceptable, especially for an intraday model.

    Without giving away the method, what is the market movement you are trying to capture?

    Mike
     
  7. Seems you already know the answer to this is an emphatic "Yes!"
     
  8. There are folks who generate (and pick from) random systems. Those are the only ones I know of who do not curve fit.
     
  9. very good question.
    the one not curve-fitted at all is the system which generates no profit over the long run.
    the more profit or the more consistent returns it produces - the more it is fitted to the curve.

    yet 99.9% of traders are unable to grasp this simple concept!
     
  10. You didn't curve fit your system.

    You changed the reward:risk parameters from negative (double the amount for loss as there is for the potential gain) to 1 to 1 (same amount for loss as there is for potential gain).

    Now, provided your system trades with positive expectation, you should do well.
     
    #10     May 4, 2009