Back Testing Technical Startergies

Discussion in 'Strategy Development' started by Soros, May 15, 2007.

  1. Soros


    Their is a lot of literature debating weather technical analysis really works or not and if it does which indicators or theories are useful and which are just black magic? One of the best ways to address this question is to use today’s powerful computers together with specialized software and test the indicators, theories and strategies that have been developed and published over time. Over a period of one year together with a colleague we tested a number of renowned as well as some not so famous indicators and strategies. This was done using Metastcok Professional 9.0 as well as Trade sim, add plug in software written for Metastock to analyse the statistical results. The strategies were tested on US stocks based on 20 years of historical data from AMEX, NASDAQ and the NYSE. Indicators tested included Moving Averages, MACD, RSI, Stochastic, Volume, McClellan Oscillator & Summation Index, Candlestick Formations, Range Beak Outs, Volatility Breakouts, Support Resistance, Rate of Change or Momentum, Relative Strength, Average True Range Stops and Gaps among others. The data was then compiled and the parameters for the variables were optimised. The problem of optimisation and curve fitting was recognized. A way that we used to get around this issue of curve fitting was by picking the best parameters for variables, which also had good parameters around them. For example if the 35 day exponential moving average worked well but the 34 and 35 day didn’t, we would conclude that this was not significant and looked for other parameters which were. We also ran a backrest for the 20 years as well as an out of sample test for around 6 months. The results we obtained were very interesting and in lighting. After having read many technical analysis books and having seen such strategies advocated by authors who proclaimed them to be reliable, it was interesting to see which indicators and strategies actually worked in the past and possibly had a chance of working in the future. As opposed to which indicators showed poor results.

    From our results we found such popular indicators as MACD, Stochastic and candlesticks as having little value or being wrong more times then they were right in predicting future price action. The analysis showed that based on are algorithms and the criteria specified accepted indicators such as Moving Averages, ROC and RSI had returns and draw downs that were unimpressive, but in general could have been utilized in a profitable manner over the long term. Further more from our analysis we found that pretty much most indicators did poorly to average when used by them selves. When indicators were combined with other such technical indicators or specific entry and exit rules based on price, we started to obtain some attractive results. Indicators when combined together with others or specific price action proved to be quiet valuable. With out going into too much statistical detail indicators such as Volume, Momentum, RSI, Range Break Outs, Support Resistance, Rate of Change or Momentum, Relative Strength, Average True Range Stops showed promising results when combined into to specific groups or with the right buy and sell criteria. Indicators such as Gaps McClellan Oscillator & Summation Index and Volatility breakouts gave mixed results. Gaps worked well if you were able to get in at the open of the gap day and predict weather the day would end up being a true gap day rather then one, which moves through the price of yesterdays high. Unfortunately this is not possible in real life. The indicators and strategies that worked best in back testing also ultimately proved to work best in forward testing. Although the results were less impressive which was to be expected they still proved to be statistically significant.

    Some of the problems in the tests included using specific entry and exit rules. Different rules or other criteria if added may have improved some of the results of the poor performing indicators and strategies. Also most likely not all the curve fitting was eliminated an indication of this is the forward test results being worse then the ones in the back tests. Other criteria such as market capitalization were not considered since they are not part of the technical analysis world but we believe could have important implications for the use of such strategies. Based on are results we consider technical analysis as a valid form of analysing and trading the US stock market and quiet possibly other financial markets. Further more we observed that if the right indicators are combined as well as specific buy and entry rules it is possible to create high return generating strategies. Overall the testing was very in lighting and a lot of knowledge was gained in regards to some of mainstream as well as less used technical indicators and strategies in technical analysis.
  2. Can you provide more data about how the indicators were used? You said MACD wasn't profitable by itself. What criteria did you use with it, for example?
  3. Did you impose a rule to exclude combinations of low volume stocks and high trading frequencies?

    What about stocks currently listed on Nasdaq, but listed as OTCBB 15 years ago?

    Did you include stocks like Enron?

    How did you account for the fact that when running multiple tests, you can expect some to show positive results just by chance?

    Something related was done 7 years ago by Katz and described in the "encyclopedia of trading strategies". I remember he found some interesting results when using a genetic algorithm to optimize multiple trading rules.
  4. I like MACD. If MACD shows negative expectancy over time, just trade anti-MACD and it should give you positive expectancy :D
  5. MACD only works for long trends because it gets you in very late and out very late
  6. It's extremely important to remember that when something is tested and it has value or doesn't have value...

    The results is based upon your methodology.

    Simply, for example, you may be using s/r levels one way while the other guy is using s/r levels via a completely different entry and trade management criteria.

    Therefore, its not really a problem...

    It's just reality and such explains why traders using the same thing will get different results because its being applied differently.

    Just as important, in my opinion, longevity as a profitable trader is dependent upon that TA should be just one of your trading tools and not used alone to make trade decisions.

    There needs to be an understanding of the price action prior to TA application.

    That reason alone is why profitable traders using TA are also using other tools with the TA instead of trying to use TA to define the price action.

    With all that said, I don't think any testing results is valid (good or bad) unless the tester reveals the specific criteria or trade management parameters.

    Thus, there are those that say it works without any proof and there are those that saids it doesn't work without any proof other than saying...

    I've tested it.

    (a.k.a. NihabaAshi) Japanese Candlestick term
  7. ER9


    lol...serious......were there any really bad performing indicators? i would love to see one that was only 25% successful with heavy negative returns. i would love to trade the opposite side of that indicator.

    maybe the study should be redone now that the preliminary data is in.....for all the statistically significant indicators that performed poorly....i wonder what the outcome over the same period of time if the opposite signal was taken?
  8. dpiatek


    An indicator is only as good as the coding of the strategy built around it. Because of this I do not believe one indicator can be seen definitely having less value than another. Perhaps if traders use these indicators in their standard form? In my experience, any indicator having at least some value is either one completely designed yourself, or one in which was modified from its original form to suit the purpose of a specific application. And in neither of these situations does using just one indicator constitute a "great" system for generating buy and sell signals. Again this is just from personal experience and in no way do I believe a "one size fits all" approach applies to trading.

    Best Regards,
    David Piatek<br><a href=""></a>
    Intuitive Financial Software
  9. ER9


    it seems to me that we are all in search of the best performing indicator/combinations of indicators in search of that statistical edge...i wonder if anyone ever did a search/study for the poorest performing indicators that consistantly provided statistically significant negative returns? maybe we are looking in the wrong direction for that statistical edge. it seems many traders are very good at loosing money....maybe we should focus our search more on what were good at :D
    #10     May 16, 2007