I am trying to develop a strategy for trading, and I want to get some opinions on what some of you think about what constitutes "sufficient" backtesting. I will supply my own ideas and perhaps that will help elicit some feedback. How many years back to test? I think roughly 5-7 years is good, since that is "recent history" in terms of the equity market trading patterns, given new technologies etc. Any difference of opinion on that? In my opinion the best way to gauge the success of a system is to give it an "imaginary" money account, let it trade that account, and the more money it makes, the better it is. Other methods may include percentage of winning trades, or average gain per trade (win or lose). Any opinions from those who have developed successful strategies? As far as accounting for all the variables, I figure give my trades 0.07 per share slippage, always try to use intra-day data if the trades are short, and don't let my simulation make trades so large that they disturb the market price if actually traded. And account for commisions of course. Anything missing? Any disagreement on anything here? If using adaptive optimization, you should always do "out-of-sample" testing as the measure of success. Anyone have more to say about the sufficiency of backtesting? What have you done to verify the soundness of a successful technique before implementing it?
there should be a FAQ... 5-7 years don't come close to covering a complete market cycle. the only true out of sample data is the data you don't yet have. not everybody agrees with the above.
If a trading algorithm is "trained" on 1000 stocks, and is afterwards "tested" on an additional 1000, the additional data is something the algorithm has not yet "seen." The first 1000 make the sample, the second 1000 make the "out-of-sample," and almost by definition the out-of-sample data is data that is "unseen." So what would you put in the FAQ.
for the FAQ? that what you suggest in splitting the field of stocks from one timeframe in half doesn't even come close to meeting out of sample testing requirements since most stocks, most of the time, are moving in the same direction. all you're doing is testing "A" against "a lot like A". further, that in following such an approach you extremely quickly run out of data for the inevitable design iterations since you can't use any data twice. for starters.
What you say in your second post makes a lot of sense... however I still fundamentally disagree with what you say in the first post. What I provided to you was an example, easy to criticize when you have a level of expertise that you now are demonstrating, but provided to one that did not appear to have that expertise. Why not share more?
Oh yeah, one more thing I wouldn't mind your opinion on, since you're interested enough to discuss it... How much different in your mind should "A" be from "a" to be properly considered "out-of-sample?" For example, I'm not going to train my stock-trading algorithm on seismic data. But could I use it for testing as "out-of-sample" data? Seems ridiculous doesn't it. However, seismic data can be viewed as a chaotic system... as can the stock market. Are they now similar enough to be comparable? Can I use one as "out-of-sample" data for the other? I certainly don't think so and I don't think anyone would really disagree with me on that. I think it goes to show that "A" has to be quite a bit more like "a" than seismic data to qualify, but how much is too much in your opinion? How similar is too similar? Is it simply enough to go one step further and segment the data by timeframe? Or are stocks too similar in general... do I need to use FOREX or futures data instead? Just would like to know where exactly the line is drawn, since you seem to know. Thanks in advance!
As is typical on this site, the self-proclaimed "experts" have responded to you without first bothering to ask the most important questions, which are what timeframe are you trading, and what instruments are you trading? Are you daytrading or swing trading? Trading equities, indexes, currencies? And what generally is your strategy? That will help determine how you need to approach your backtesting duration.
you're asking the wrong person. if you look back at other discussions i've had on ET you'll see that - as far as i'm concerned - there is no true out of sample data except that which has yet to be generated. the entire premise of walk-forward is IMO flawed (when it comes to the market) and you're better off just test on the entire data set at once and spending the time in frame-by-frame analysis to see where the hiccups happen. this is not a popular position on ET.
What amuses me is you guys keep on beating this horse to death. Frankly, institutions go through phases as regards both development and testing of programs. At this point in time, not many are using in sample/out of sample testing that you talk about so much. The reason why? Not effective. So what I am trying to say, is that the professionals have moved on. At this point in time, they are approaching the problem from another direction. I don't want to bore you with the details. What I willl say is that there have been a few posts relating to how fit a program to a market and vice versa. The posts were largely ignored. I assume that is because to make use of the info, you would need to have a background in statistics and programming. This is probably why institutions are willing to pay people who graduate from top 10 schools with PHds in math, statistics, computational methods and computer science so much money. As far as my own motivation, well I was lucky enough to have a good programer give me a heads up. I figured I had a decision to make. Get educated or pay someone else to do the work. I think you folks have the same decision to make. Good luck tomorrow. Lefty