General Statistics Question

Discussion in 'Strategy Development' started by HotTip, Aug 30, 2011.

  1. HotTip

    HotTip

    I have a question for the stats studs out there:

    If I developed a system that when backtested creates a smooth upward daily equity curve (i.e., >99% correlation) incorporating >3611 trades over 427 days, is it even necessary to do an in-sample/out-of-sample test on that? I can understand doing that when the in-sample results have a high beta and generates an erratic equity curve, but if my equity curve has a nice upward trajectory regardless of the time period I'm looking at, doesn't that already imply that the system is consistently profitable, or is an out-of-sample test still necessary regardless of the equity curve my in-sample generates?

    Thanks in advance for any insight!
     
  2. On what instruments? On what timeframes / volume / tick ?

    Not to rain on your parade, and I have no idea what you are doing, but 427 days is meaningless IMO.

    Others may refute that.
     
  3. rdg

    rdg

    It's not a matter of whether it's necessary or unnecessary, it's a matter of how confident you are that the feature you are exploiting exists in any data other than what you are currently testing against. For example, if I have a development process that starts with a bunch of data and finds a system that trades it perfectly, then I'm not going to have much confidence in it. I will probably want to see how it performs on different data. On the other hand, I could also imagine a situation where I had high enough confidence, based on the rigor of my development process, to take a system live after seeing only a few test results.
     
  4. maler

    maler

    All my backtests use some form of walk forward analysis.
    Do not trust in sample backtests.
    Whenever I see a very smooth equity curve,
    the most likely explanation is some kind of mistake.
    Most common mistakes are look ahead and suvivorship bias,
    or unrealistic execution costs.
     
  5. kickout

    kickout

    as a pseudo qualified statistician....there is no question there...



    analogous to randomly asking somebody "is this important"
     
  6. toc

    toc

    what is the win loss percentage on the number of trades in the backtested system.