This is a good book but older content. There is a lot of talk about Aronson's book in several quant bolgs: http://www.amazon.com/exec/obidos/ASIN/0470008741/autotradblog-20 Some people love it and some hate it. For sure it raises the subject of trading system development to a new level. The most important chapters cover the statistical analysis of backtesting results using boostrap resampling and Monte Carlo permutation methods.
Profitability and Systematic Trading includes about 40 pages of pattern code some people may not find useful at all. But there are 136 single-spaced pages packed with information with just a few charts displayed. Harris is an advocate of automated trading system design and strict risk and money management. He thinks that spending time to back-test random ideas is a waste of time and money and people should instead focus on automating this task and controlling risk. I agree about his software. It is one of the best investments I have ever made in a trading tool. He has a lot of free information though and IMO one can benefit a lot by following his blog postings, especially his recent ones about ETF trading: http://www.priceactionlab.com/Blog/2011/01/the-state-of-etfs-january-16-2011-weekly-recap/
Well, I am verifiably a feeble-minded old fool, but I quickly scanned your reference, and I don't understand why conventional backtesting does not qualify as scientific method. You hypothesize. You run experiments (backtesting). You modify your hypothesis. Repeat. And home in on a codified statistically valid theory which is extensible.
Three great books I havn't seen mentioned yet: The Encyclopedia of Trading Strategies- Jeffrey Owen Katz Trading Systems that Work- Thomas Stridsman Trading Systems; Secrets of the Masters- Joe Krutsinger These are good(although a bit dated) basic books that can give you some insights into developing your own system. Goodluck... GDog
Harris does backtest. However, apporach he advocates is to avoid manually picking patterns for a trading system and instead automate search for best patterns. Because traditional trading system qualities such as percentage of winners, profit factor, Sharpe ratio and many others don't say anything about how likely it is that a trading system actually works as opposed to be purely reesult of overfitting. Using Harris's approach it is all too easy to create a trading system which relies on a large number of patterns. The system may have great performance statistics in the backtest. However, each pattern may appear in the whole data only a couple of times, which is absolutely insufficient to determine if the pattern works. Such a system has no potential to perform in real world but on paper it will look great. Finding a system that has good backtest results is effectively a datamining exercise. So, distingushing between good results and bad results is far from trivial. Read more here.
Ah, thank you for that concise cogent summary. I will offer a wholly personal view on that approach. I have found in my own practice that the statistical validity of a tested hypothesis works better in live testing if you do the following: form the hypothesis based on apparent repeated observation of price action, like "price tends to break out of the clean side of a consolidation rather than the raggedy side" (not a valid hypothesis, BTW) make sure that the hypothesis has some valid explanation in market principles, like "the big boys are holding the line until they have discouraged the weak hands" run an initial test based on eyeballed estimates of likely loss and profit stops, and proceed if this has positive expectancy optimize the stop loss with no profit target, and proceed to profit optimization if the best stop loss is weakly profitable or weakly unprofitable test far enough back in time to get supposedly statistically significant sampling, but not so far back that volatility variations confuse the issue do rolling tests of the same time duration as time proceeds and estimate the stability of the hypothesis as volatility changes. I believe in doing this going forward, not backward in time, as a paradigm shift may have occurred in the past, and you want to be au courant. P.S.: Thanks for that link. And I forgot to add that one requirement in hypothesis formation is that it helps sometimes if the hypothesis is ridiculously absurd.
Way of the Turtle, by Curtis Faith (a former Turtle) is oriented toward longer-term systems, but has a lot of good insight and ideas for system development in any time-frame. There is very little space devoted to actual coding, so other books would be necessary to develop the actual "plumbing" behind the system. Some may view the book as a shameless marketing brochure for his software, but I think that the concepts can be adapted to any software. (Note: I don't recommend his later books)
I have been using the Harris method for the last three years. I agree it is hard to develop a system that will be profitable not only on paper but I don't see that as a flaw in the method but as a failure to test the patterns properly out_of_sample and to use them in non-trivial ways. The Harris method relies on the fact that there is clustering of patterns which reduces data mining bias due to correlated rules. However, if someone expects to use 3 months of 5-minute data and find a system that works using the Harris method or any other data mining method he will fail. Data mining bias is inversely related to sample size. So just to make a long story short the following are necessary for the Harris method to have significance in my own experience: (1) Large sample size (2) proper out_of_sample testing He has some examples of forward tests in his website. Scroll down to example 2 where he first selects 17 patterns with more than 50 trades from the in_sample results and then forward tests them. Only 4 out of the 17 fail in out_of_sample. That was for position trading system for SPY with 2% stop and it is quite unlikely that the market would be affected by the execution of the system.
speaking of curtis faith, the turtle trading rules he put together I believe are a must read for anyone starting out. A great primer that covers what should go into designing a mechanical ystem. https://www.bsp-capital.com/documents/turtlerules.pdf