Just how reliable is backtesting?

Discussion in 'Trading' started by nxt7, Apr 14, 2016.

  1. lindq

    lindq

    You don't know if your system is in a drawdown, or broken.

    The fact that you don't indicates a lack of confidence in your system.

    You simply have not lived with it long enough to make a judgement. That ONLY comes from experience.

    You may have something valuable. Or may not. Only time will tell. No shortcut for that.
     
    #21     Apr 14, 2016
    Xela and d08 like this.
  2. Xela

    Xela


    Personally, I trade from price action, without indicators. But that has nothing to do with the discussion, really ... I'm not suggesting that "nobody should use indicators" just because I prefer not to.
     
    #22     Apr 14, 2016
  3. Simples

    Simples

    Try to list your assumptions, then what might possibly break them.

    How long is your backtest? What is the reasoning behind your trade plan? If it's just an MA crossover system, why do you expect consistent profits (edge) from using an average of prices as triggerline? If this instrument had stellar historical results, why wouldn't it return to the mean of the other instruments now? Where in the business cycle is the market now, and are you playing the right direction or waiting out possible adversity? Do you have enough trust in your system, robustness and financial stamina to keep at it, or may that mean you go bust? If you forget everything you think you know and just look at the long-term chart, what might be happening right now do you think and why would your system outsmart the market?

    This is a hard simple game. Are you prepared to double down on time and avoiding spending too much on dead ends? What's keeping you from seeing what happens down the road (several months from now)?
     
    Last edited: Apr 14, 2016
    #23     Apr 14, 2016
  4. qxr1011

    qxr1011

    The main problem with back-testing is not just how well its results, based on the past data, will indicate the future performance of the method.

    The main problem of the back-testing is that it does not test the trader himself !

    So when in the future the day will come and the shit will hit the fan, the method may still be working, but the trader (who was never tested) will start pissing in the pants and abandon the method.... :)
     
    #24     Apr 14, 2016
  5. userque

    userque

    Backtesting is extremely reliable if done properly. The backtesting should trade exactly as you would trade forwards.

    There's much more to it, but two main points are proper use of out-of-sample data and avoiding leaks from 'future' data.
     
    #25     Apr 14, 2016
  6. A good, informative interview on "backtesting" systems.

    Is it skill or is it luck ...
     
    #26     Apr 14, 2016
  7. zdreg

    zdreg

    sounds like you learned from the author of the master algorithm
    http://www.washington.edu/news/2015...domingos-author-of-the-master-algorithm/:cool:
     
    #27     Apr 14, 2016
  8. Good interview, but his book is not really selling that well.
    However, this guy's book is:

    Wow, 247 reviews !! 5 star average rating !!!
    This guy must be on to something....
     
    #28     Apr 14, 2016
  9. Surgo

    Surgo

    Err... "Cool in a crisis? UW Emergency Management seeks volunteer responders"? Not sure what that has to do with anything. I'm just talking about basic statistics concepts that you'd learn in the first week of any college-level machine learning class.
     
    #29     Apr 14, 2016
  10. zdreg

    zdreg

    #30     Apr 14, 2016