better way to do Out-of-sample test?

Discussion in 'Strategy Development' started by mizhael, Apr 18, 2010.

  1. How do you decide the in-sample and out-sample sizes (two extra parameters) in Walk Forward Optimization without being curve-fitting?
    #11     Apr 20, 2010
  2. the best method of avoiding curve fitting is not having inputs at all.

    if you want to optimize then you should do it properly. make a plan with your available data and split it up according for in and out of sample runs.

    1. Optimize over X period
    2. Run setting over out of sample
    3. Repeat

    there are no shortcuts.
    #12     Apr 20, 2010
  3. Unless your strategy is dynamic adjusting to market changes of course your going to have different results.

    ES trading at 1200 - 1500 is very different from ES trading 600 - 900.
    Option strikes are still 5 points but the contract notational value is $30K vs $75K.

    How well does your strategy adjust, manage risk and perform trading random numbers? After all that is really what your after.
    #13     Apr 20, 2010
  4. Incorrect. This applies to everyone's strategies.
    #14     Apr 20, 2010
  5. It all has to do with whether you found something, that given the current market structure, has a REAL

    99.9+% of things backtested will prove to be spurious. And most of the things that are walkforward test will still fail.

    The size of the sample is to give SOME statistical credence to the idea. Too small a number or too short a time means you have less confidence in it.
    #15     Apr 20, 2010
  6. for sure ...thats the reason for walkforward.. if done correctly it will catch the contracts cyclical (or other ) changes . I use 6 years of tick data (and percentages) to check the robustness of my ES parameters.
    #16     Apr 20, 2010
  7. Careful... last trade time & sales tick data will lead you down a false path. Test against bid/ask/last trade data or better yet level II tick data if you plan to trade large lots.

    #17     Apr 20, 2010
  8. I have a system I developed in 1993 and it still makes money.

    All universally qualified propositions, like yours, are proven false in the face of a counter-example.
    #18     Apr 20, 2010
  9. vikana

    vikana Moderator

    I eventually settled on a different way to optimize. In my opinion the point is not to "optimize" but to "stabilize" and I therefore approach the problem as follows:

    I break down the data in e.g. 1 year blocks and optimize each of those blocks separately. I am looking for consistency across the blocks. If a parameter drifts, i look for a different way to express the relationship I'm optimizing so that I can get the stability I'm looking for. I have had very limited success with systems where the parameters needed frequent optimization (but that's just my experience).
    #19     Apr 20, 2010
  10. Not can account for slippage. You can not get (that I know) 6 years of bid/ask/last data. I trade large lots and monitor bid/ask sizes.
    #20     Apr 20, 2010