Overfitted? Please help take a look at my PNL curve...

Discussion in 'Strategy Development' started by mizhael, Jun 11, 2010.

  1. Here is it:


    The vertical line at around March 2010 divides insample and out-sample.

    The Outsample looks more choppy than insample...

    This is a trend-following strategy on commodity futures...

    I am afraid I got overfitting...

    Any thoughts?

    thank you!

    (I normalized)
  2. You do not have enough of out_of_sample to draw any conclusions but I believe you it is overfitted, can't be otherwise.
  3. Thanks for your comments.

    When and how can I judge it's overfitted or not?

    I hope I can find a way to convince my collaborator that it is overfitted and we should shut it down.

    We've lost a lot of money recently...
  4. It depends on how you developed the model.

    1. Did you have a conceptual idea of what your were looking for in terms of market dynamic? Was it a well known market dynamic?

    2. Did you create a "simple stupid" backtest with no fitted parameters to test the initial concept for validity?

    3. Did the "simple stupid" model produce ok results on similar products?

    4. Did you properly simulate costs and fills?

    5. Do you understand why the model went into drawdown during certain periods in the backtest?

    All the above points matter quite a bit and any flaws in any one of the steps above will cause inconsistent results going forward.

    If you did 1-4 correctly and then did some optimization, what exactly did you optimize and how?