Just how reliable is backtesting?

Discussion in 'Trading' started by nxt7, Apr 14, 2016.

  1. Sergio77

    Sergio77

    Not even close. One major issue with backtesting results is not the number of parameters or curve-fitted vs. over-fitted but is related to how many variations of systems you tried before the one you finally chose. Read this article that I think explains the issue well and in graphical form.
     
    #61     Apr 16, 2016
    Simples likes this.
  2. Sergio77

    Sergio77

    True. This is a very important paper on this subject. A must read.
     
    #62     Apr 16, 2016
  3. userque

    userque

    For shi+s and giggles, I did this using logic that dynamically adjusted the MA parameter based upon the recent price action. The EA determined the optimal factors associated with the formula to alter the parameter.

    This simple, single-dynamic-MA-crossing-the-underlying system was profitable.
     
    #63     Apr 16, 2016
  4. kut2k2

    kut2k2

    I don't believe there's any single system that works in all market conditions. A combination of systems would be required, one for trading trends and another for trading in choppy conditions. And a regime switch of some kind to determine when to use which system.

    Something like

    http://www.elitetrader.com/et/index.php?posts/2043606/
     
    #64     Apr 16, 2016
  5. benwm

    benwm

    Did you trial CATSCAN (Choppy And Trendy Scanner)? Any experience with it, or did you build your own version along these lines?
     
    #65     Apr 16, 2016
  6. A good system consists of subsystems that each are optimized for specific market behavior. And there is a switch between the subsystems depending on the trend. I wrote:Your system should be self-learning or self-adapting.
    One system can have 1000 subsystems which make all together one system.

    I trade one system that is selfregulating, which means that it adapts itself to the strenght of the trend. In a choppy market it takes small profits and when the trend is strong it can stay in for hours. I am daytrader.
     
    #66     Apr 16, 2016
    dartmus and kut2k2 like this.
  7. kut2k2

    kut2k2

    I have no experience with CATSCAN.
     
    #67     Apr 16, 2016
  8. Humpy

    Humpy

    When back testing on MT4 I notice that on some forex pairs they seem to work well but on others they makes losses. I suppose this indicates weaknesses of the EA and should be avoided.
     
    #68     Apr 16, 2016
  9. d08

    d08

    It's not a requirement but it can be beneficial. I run everything all the time, every system needs to at least produce a flat curve in adverse conditions. I've yet to find a reliable method to distinguish between trending and ranging - one without lag.
     
    #69     Apr 17, 2016
    Xela likes this.
  10. userque

    userque

    Clarification:

    I've assumed that people using "EA," were referring to "Evolutionary Algorithm." I now see that many here are using MetaTrader and EA likely means "Expert Advisor."

    I don't use MT and my EA above actually means evolutionary algorithm.
     
    #70     Apr 17, 2016