tntneo - Agree with you, except perhaps the part about "out of sample" testing on other markets. In essence, that's what the forward test is (i.e., the forward test time period is by definition outside of the backtest sample). As far as having to test a system on other markets though - I don't see that as a given. It would certainly be ideal if you developed a system that worked as is in a lot of different markets, but it's by no means a necessity. I've seen a number of people on ET talk about backtesting/optimizing systems on a thousand stocks or dozens of futures contracts, but there's nothing wrong with having a system that works well on one market. There are lots of private trading companies out there using mechanical systems that are specific to a particular market (i.e., one system to trade the S&Ps, another to trade the NDX, another for Crude Oil, etc.). Afterall, if you've got a verifiable consistently winning system that works on the S&Ps, would anyone trash it just because it's a dog trading Crude Oil? Even at the extreme, would anyone blow out a system that generated consistent winning results only on MSFT?
ArchAngel, I have a slightly different view about the out of sample. However, I like to make a run before the sample data because the forward testing may be close to the last data tested. regarding the multiple markets I fully agree with you. My remark was an advice for people starting with system trading. It is important to realize that a system works in a specific market condition and/or market, while maybe the designer did not realize it (thinking the system is just great. which is what the thread's author said). I did build systems for single markets (futures) and knowingly would never trade the system on another market (even correlated). Each market has its personality (well a main one and then change behaviour from time to time). So of course, there is nothing wrong building a system for a single market. it's easier, it usually last longer but you can milk out hard for that time. building for a single market is just a risk of overoptimization and think it is 'fantastic' while you never make a dime with it. tntneo PS : overoptimization is the trouble. try hard to know when you are doing it. checking multiple markets is a way (it does not have to work, but you should know why it does not). that was my advice actually.
I agree with the reply that said, "What are the chances that SUNW, MSFT, etc will trade in the same manner as they did in the 1990's and early 2000's". That is exactly the problem...To some extent you almost have to forecast the trading environment and then create either strategies of systems that will perform in those environments...And many people will tell you that you bread and butter system in a bull market is your death bed in a bear market, mainly because the action is different and the trade management is much different as well... I think the most enlightening aspect of system testing is just testing the simplest n-breakout strategies and trying to trade around these types of strategies...WHen I do this sort of testing for the index futures it gives a better feel for where the bigger money is coming and going...
Sounds like you are simply data mining Having found a 600% gain in the past ain't gonna necessarily lead to similar things in the future. Now, if you had an idea in mind before going and sifting through the data, that would be different. That's my own opinion though....
Vlad, We really liked this approach when I was emplyed to do mechanical system research/development. At the time, we gave up on it and decided instead to "punish ourselves" (had a Utility function) everytime we ran an optimization/walkforward so that we could not get too acquanted with the data. We finally gave up at the time. Today, that is how I "started" - what a difference! nitro