backtest for 3 years, blow up in 3 days,

Discussion in 'Risk Management' started by jacksmith, Mar 23, 2009.

  1. The only thing that would cause a system to blow out in three days is absurd position sizing and poor risk control.
     
    #11     Mar 24, 2009
  2. Do you realize you just gave away your "edge." Now you will have to start over.
     
    #12     Mar 24, 2009
  3. Your system could blow up for any number of reasons. Some of which are:
    - Most systems have just as much of chance going into a drawdown as a wining streak to begin with.
    - The time period of 3 days is irrelevant. More time may be needed to get a significant amount of data to draw conclusions.
    - The current market conditions do not match what you tested with. Are price direction and volatility in the same realm as in your back testing?
    - You system is curve fit and is not relevant to changing data.
    - There was insufficient out-of-sample testing done to draw any conclusions.
    - You did not paper trade the system so stats could be compared to the original back test.
    - With the current price data are you bypassing logic that was used successfully in back testing?
    - Are risk and money management performing in live trading the same as you back tested it?
    - Are entries and exits where you expect them to be?
    - Which stat is out of line from the back test to live test? Larger losses? Smaller profits? Bigger drawdowns?

    These are just of the few items to look at.

    In building systems for the last 12 years the normal problem I find in systems is with price and volatility adaptation. This means the optimized values build a solution that is custom to the price data at hand (curve fit) and will not work with new data presented to it.

    Is there a solution to stop curve fitting? I will tell you one way that has been passed around for years. But it’s a lot of hard work. About 10 years ago at a trading show workshop I heard some one say “The trick to building systems is not to optimize them. That means to keep optimized values constant and vary the code until it performs well….”
     
    #13     Mar 24, 2009
  4. And then what?
     
    #14     Mar 24, 2009
  5. heech

    heech

    Have a beer? Turn on family guy?

    The idea is just to take "out of sample period" (thanks to earlier poster for the term) data, and see whether your strategy still holds up.

    If it doesn't, then revisit... just be aware that if all you do is tweak your strategy to show good results, you're just curve-fitting again. You should test again with a new series of "out of sample period" data before going live.
     
    #15     Mar 24, 2009
  6. Stupidity
     
    #16     Mar 24, 2009
  7. That's the problem - you CAN'T revisit, even with a completely unrelated strategy, or you've started of your next strategy testing with internalized knowledge of ALL the historical data.

    Which means you're curve-fitting from the get-go.
     
    #18     Mar 24, 2009
  8. Datamine vs dump your variables into a computer and let it spit out your roadmap. That will never work and you must have some trading edge or some dynamic feature that will adjuts to the changing environment. Plus the knowledge of your systems and what environment it will not perform well in and try to filter or pause during this condition.

    It ait easy and you will put in your 5000 to 10,000 hours before you see anything robust and not just random luck.
     
    #19     Mar 24, 2009
  9. gkishot

    gkishot

    There is only one reason that might cause this - leverage.
     
    #20     Mar 24, 2009