in sample and out of sample testing

Discussion in 'Strategy Building' started by zordan, Mar 17, 2013.

  1. zordan

    zordan

    I have been looking for a while for vendors that show some out of sample testing for their systems.

    This example seems pretty good to me. What do you think of this system? It it worth a test?

    http://www.vbosystems.info/videos.html
     
    #1     Mar 17, 2013
    Share
  2. dom993

    dom993

    I just glanced at their backtesting results, their systems have between 200 & 600 trades over a 6 years period, this isn't enough for me to get interested ... to easy to have (intentional or not) over-optimization on such a small sample set.
     
    #2     Mar 17, 2013
    Share
  3. zordan

    zordan

    good point. what would you consider a minimum number of trades for 1 market futures system over 1 year?

    did you check the video? how could it be over optimized if the out of sample performance is as good as the in sample one?
     
    #3     Mar 18, 2013
    Share
  4. dom993

    dom993

    I no longer bother with anything less than a few thousand trades in backtesting, and minimum 5 years back (preferably 10 years).

    WRT out of sample testing, it doesn't mean much to me, as obviously every one discards systems with bad out of sample results, so this is no better IMO. The only thing that counts beyond backtesting is live - audited - results.
     
    #4     Mar 18, 2013
    Share
  5. zordan

    zordan

    thanks for the clarification.

    the only issue i see with live, audited, results is that you need to move from the space of "systems vendors" to the space of registered CTAs (commodity trading advisors).

    there are still 2 issues i see:

    1. for registered CTAs normally starting capital minimum are higher... almost always over 100K
    2. the comment you made about survivorship bias for outofsample results still applies here: non performing CTAs go broke so any list you see suffers by definition from survivorship bias

    i am still looking for a honest system vendor that offers decent systems in the retail space that i could add to my list.

    if you know any i'd be interested in knowing your experience.
     
    #5     Mar 18, 2013
    Share
  6. RobertG

    RobertG

    Any systems that is sold to the public, and has no control over the number of lots traded will fail.

    Get a good system from someone who actually trades or not available on the shelf for everyone to grab. Try your broker, some do offer good systems not available for the public to "buy" and would lease only.

    RG
     
    #6     Mar 23, 2013
    Share